Slide 1
Slide 1 text
2019 DevDay
Data Science Drives Improvement
of LINE Messenger
> Taro Takaguchi
> LINE Data Science Team2 Senior Data Scientist /
Manager
Slide 2
Slide 2 text
Agenda
> Introduction: Data Science Team
> Challenges in LINE App Improvement
> Improvement of “Create Group” Feature
> Data Science Tools
Slide 3
Slide 3 text
LINE App Improvement Project
Data Driven
Diverse
Team
Users First
In-house
Development
Slide 4
Slide 4 text
Introduction: Data Science Team
Slide 5
Slide 5 text
Organization
Data Science
Team
Engineering
Infrastructure
Data
Platform
Data Science And
Engineering Center
GROWTHY
Platform
Data Labs
Machine
Learning Team
Slide 6
Slide 6 text
Coverage of Services
Data Science
Team 2
Data Science
Team 1
Data Science
Team 3
Data Science Team
Data Science
Team 4
Stickers
Slide 7
Slide 7 text
Company-Wide Collaboration
Data
Science
Service
Plan
Data Science Is a Part of Interwoven Team
Client
Development
Server
Development
UI / UX
Design
Legal
Info
Security
…
Machine
Learning
Data
Platform
Slide 8
Slide 8 text
Cycle of Projects
User Research Development
Test
Feedback
Plan
Slide 9
Slide 9 text
User Research
Learning Inside & Outside of Log Data
Focused Interview Online Survey Dashboard Monitoring
Slide 10
Slide 10 text
Plan & Development
Slide 11
Slide 11 text
Test & Feedback
Online A/B Tests in
our Team (2018 -)
20+
Metrics Monitored
in a Test
100+
Users Targeted
Globally in a
Typical Test
10M+
> Online A/B testing in principle
> In-depth data analysis of results
Slide 12
Slide 12 text
Challenges in LINE App
Improvement
Slide 13
Slide 13 text
Many People Use LINE in Different Ways
MAU
(4 Main Regions)
164M
Message Types
20+
MAU (JP)
82M
Slide 14
Slide 14 text
No Single KPI
Slide 15
Slide 15 text
Core Value of LINE App
Slide 16
Slide 16 text
Closing the Distance
Easy and Pleasant Communication With Close Friends
Slide 17
Slide 17 text
Local Friend Network on LINE
Me
Slide 18
Slide 18 text
Diverse and Active Social Contexts
Family
School
Slide 19
Slide 19 text
Group Feature
Slide 20
Slide 20 text
Improvement of “Create Group”
Feature
Slide 21
Slide 21 text
Screen Order Was Not Intuitive?
> 1. Set group icon and name
> 2. Choose members to be invited
User Research
Slide 22
Slide 22 text
Swap the Screen Order
> 1. Choose members to be invited
> 2. Set group icon and name
Plan
Slide 23
Slide 23 text
No Difference
> Lift in success rate of group creation
Test
JP TH TW ID
No statistical significance
0
Slide 24
Slide 24 text
Why Made No Difference?
Slide 25
Slide 25 text
Dimensions of Analysis
> Uniqueness of each region?
> Time-dependency?
> Success users? Failed Users?
> …
Feedback
Slide 26
Slide 26 text
True Pain Point: Choosing Members Quickly
JP
TW
TH
ID
Not proceed to “group icon and name” screen
Feedback
> Where failed users gave up?
Slide 27
Slide 27 text
Potential of Further Improvement
Fail Group Creation
On the Last Day
Never Use Group
Creation Before
Use Group
Creation Before
Succeed Group
Creation Before
Fail Group
Creation Before
65%
7% 28%
Feedback
Slide 28
Slide 28 text
Show Recently Chatted Friends Section
Plan
> Scenario: 1-to-1 chat → Create group to invite friends
Slide 29
Slide 29 text
Plan B
> Put everything into 1 step to shorten the process
Plan
Slide 30
Slide 30 text
A/B Testing for 4 UI Patterns?
Alphabetical + Recent Chat
2 Step
Control
ʢAS-ISʣ
Treatment 1
1 Step Treatment 3 Treatment 2
> Complete order: 6-pair comparisons
> High false positive rate / Large sample size required
Invitee list
# Steps
Test
Slide 31
Slide 31 text
Solution: 2-Pair Comparison
> Factorize effects into 2 comparisons
> Moderate false positive rate with small sample size
Alphabetical + Recent Chat
2 Step
Control
ʢAS-ISʣ
Treatment 1
1 Step Treatment 2
Invitee list
# Steps
Test
Slide 32
Slide 32 text
Feedback
> Lift in success rate of group creation
JP: T1 JP: T2 TH: T1 TH: T2 TW: T1 TW: T2 ID: T1 ID: T2
No statistical significance
0
No Difference
Slide 33
Slide 33 text
JP TW TH ID
Users Can Find Members Quickly
> Reduction in typical time to complete group creation
* statistically significant
* * *
0
Control (AS-IS) vs. 2-step + Recent chat
Feedback
Slide 34
Slide 34 text
How About All-in-One Screen?
> Users do not prefer all-in-one screen
Feedback
> Creation of 1-member groups ↑
↓
> Success rate of ≥2-member groups
Slide 35
Slide 35 text
How About All-in-One Screen?
> Users do not prefer all-in-one screen
> Creation of 1-member groups ↑
Feedback
↓
> Success rate of ≥2-member groups
> Switch ‘Create group’ → ‘Create chat’ ↑
Slide 36
Slide 36 text
How About All-in-One Screen?
> Users do not prefer all-in-one screen
> Creation of 1-member groups ↑
> Switch ‘Create group’ → ‘Create chat’ ↑
Feedback
↓
> Success rate of ≥2-member groups
> Click Done button twice ↑
Slide 37
Slide 37 text
Another Pain Point
Slide 38
Slide 38 text
Where Is “Create Group” Button?
User Research
> (B) “Add friends” menu
> (C) “Create chat” menu in Chats tab
> (A) Home tab
> (D) No idea
✓
✓
✓
Slide 39
Slide 39 text
“Create Group” Option in Create Chat Menu
Plan
> Most of users know how to create a chat
Slide 40
Slide 40 text
JP TH TW ID
More Users Create Group Successfully
Test
> Lift in the number of users completing group creation
* statistically significant
*
*
Slide 41
Slide 41 text
All Good, Really?
Feedback
> Created groups just instead of chats?
Slide 42
Slide 42 text
Good in Total
Feedback
> More users in JP created groups or chats
Group Chat Group or Chat
Lift
0
Slide 43
Slide 43 text
All Good, Really?
Feedback
> Eat user traffic to Home tab and Add friends menu ?
Slide 44
Slide 44 text
Good in Total
Feedback
> Add friends and official accounts
> Click friends recommendation
> View Home tab
→
→
→
> Click create chat button →
> No negative effects on friending
Slide 45
Slide 45 text
Next Phase…
> Increase of “active” groups
User Research Development
Test
Feedback
Plan
Slide 46
Slide 46 text
Data Science Tools
Slide 47
Slide 47 text
Steps for A/B Test Data Analysis
1. Required sample size
2. Target user slots
3. Monitoring dashboard
4. Test results report
Data Scientist Planner
Designer
Developer
Slide 48
Slide 48 text
Steps for A/B Test Data Analysis
1. Required sample size
2. Target user slots
3. Monitoring dashboard
4. Test results report
Data Scientist Planner
Designer
Developer
Slide 49
Slide 49 text
> Presto / SparkSQL / SparkR / PySpark / Markdown
> Company-wide accessible (under permission control)
Slide 50
Slide 50 text
LIBRA
> Split users into 1,024 randomized slots
> No manual operation necessary
Data Scientist
Use Slots
#654 & #987
Test Spec
Distribution Server
Slot #654:
C
Slot #987:
T
Slide 51
Slide 51 text
Steps for A/B Test Data Analysis
1. Required sample size
2. Target user slots
3. Monitoring dashboard
4. Test results Report
Data Scientist Planner
Designer
Developer
Slide 52
Slide 52 text
LIBRA REPORT
> Semi-automatic data summarization and visualization
> Tables re-usable for test result analysis
Data Scientist
Processed
Test Results
Register
Query
Slide 53
Slide 53 text
R Shiny
> Handy and flexible dashboard
> Early detection of unexpected logs
Also see Poster #P-B2 presented by Motoyuki Oki (DS team 2)
Slide 54
Slide 54 text
Steps for A/B Test Data Analysis
1. Required sample size
2. Target user slots
3. Monitoring dashboard
4. Test results report
Data Scientist Planner
Designer
Developer
Slide 55
Slide 55 text
Conflr
> R Markdown → Atlassian Confluence wiki
> OSS developed by our team member (https://github.com/line/conflr)
Slide 56
Slide 56 text
Wrap-Up
Slide 57
Slide 57 text
LINE App Improvement Project
Data Driven
Diverse
Team
Users First
In-house
Development
Slide 58
Slide 58 text
Thank You