Data Science leads overhaul of the chat menu UI

Motoyuki Oki
LINE Data Science Team2 Senior Data Scientist
https://linedevday.linecorp.com/2020/ja/sessions/3932
https://linedevday.linecorp.com/2020/en/sessions/3932

2. Role of Data Science Team Today’s Takeaway Handle Large-scale Data
3. Estimate Business Impacts Role of Data Science Team Suggest Next Improvement Today’s Takeaway Handle Large-scale Data
4. Agenda › Improvement Project of Chat Menu UI › Challenges in Analysis › Further UX Improvement › Wrap-Up
5. Improvement Project of Chat Menu UI
6. Chat Menu Renewal Released in April 2020
7. › Easy to access information in Chatroom › Settings for communication › Management of members › List of shared contents “New” Chat Menu UI
8. Why did we renew Chat Menu?
9. Chat Menu Version 0 › Legacy UI › Many icons and labels › Few clues of what is inside › Limited label length Issues
10. Chat Menu Version 1 Issues › Clicks of album and related: +30% › Clicks of events: -24% › Count of group invite: -29% › Count of friend block and group leave: -20% → Not good in total compared with version 0 (only iOS)
11. Plan for Improvement Displaying Visuals Grouping Items One Screen
12. 4 icons Candidate Plans › Take advantages of Version 0 and 1 8 icons Visual Lists & Text Items Visual Lists 4-icon UI 8-icon UI
13. Challenges in Analysis
14. How Can We Decide the Order among 4 UIs? 4-icon UI 8-icon UI Version 0 Version 1
15. Solution: Design 3 Stages of KPIs › Define KPIs in the order of priority › Apply a knockout competition on 3 stages View Click Conversion › View Count › View Days Count › … › Click Count › Click Days Count › Bounce Rate › … › Group Invitation › Note Creation › Friend Block › … First Second Third
16. How Do We Handle All 120 Patterns? 5 Room Types Friend, Group, Chats, OpenChat, Official Account 6 pair 4 Regions JP, TW, TH, ID × × › Need all-pair comparisons? › Require large sample size? 4-icon UI vs. Ver.0 4-icon UI vs. Ver.1 4-con UI vs. 8-icon U 8-icon UI vs. Ver.0 ….
17. Solution: 3-pair Comparisons › Minimize # comparisons pairs based on previous knowledge › Moderate false positive rate with small sample size vs. Ver.0 Ver.1 Comparison 1 Comparison 2 Expectation Why? Ver.0 wins Previous observation 4-icon UI wins Rich visual No idea ? 4-icon UI 8-icon UI vs. vs. Ver.0 4-icon UI Comparison 3
18. Solution: Auto Analysis by Libra Report › Provide semi-automatic data summarization and visualization › Create re-usable tables for test result analysis
19. First Stage: Draw View › View Count → › View Days Count → › … View Count ▪ Version 0 ▪ Version 1 ▪ Version 0 ▪ 4-icon UI ▪ 4-icon UI ▪ 8-icon UI
20. Second Stage: 4-icon UI wins View Click › Click Count +0.01 per user › Click Days Count → › Bounce Rate -9% › … 4-icon UI › View Count → › View Days Count → › …
21. Decision: Choose 4-icon UI View Click Conversion › View Count → › View Days Count → › … › Click Count +0.01 per user › Click Days Count → › Bounce Rate -9% › … › Group Invitation → › Note Creation -0.00006 › Friend Block → …
22. Drastic Changes Does New UI Bias Distort Results? › Increase clicks only when users visit Chat Menu for the first time?
23. Solution: Estimate Effects of New UI Bias 100 110 120 130 140 150 160 Week1 Week2 Week3 Week4 Version 0 4-icon UI › Apply difference-in-difference approach
24. Solution: Estimate Effects of New UI Bias 100 110 120 130 140 150 160 Week1 Week2 Week3 Week4 Version 0 4-icon UI › Apply difference-in-difference approach *Statistically significant decrease
25. How Much Are Business Impacts? × × 166M Click Count: +0.01 Period: 1 year Population among 4 regions › Translate observed effect size into real outcome +0.01 = +600M As of June 2020 click per year
26. Further UX Improvement
27. Is Position of Setting Icon the Best? › Users can click setting icon to change chat room settings
28. Data Analysis Finds Pain Points
29. Data Analysis Finds Pain Points › 40% of sessions: “Link” or “File” → “Setting Icon”
30. Data Analysis Finds Pain Points › +105% Sessions of “Block” → “Unblock” › 40% of sessions: “Link” or “File” → “Setting Icon”
31. Decision: Rearrange Setting Icon
32. › Easy to access information in Chatroom › Settings for communication › Management of members › List of shared contents “New” Chat Menu UI
33. Wrap-Up
34. Role of Data Science Team Handle Large-scale Data Estimate Business Impacts Suggest Next Improvement Today’s Takeaway
35. Thank You