Data Science leads overhaul of the chat menu UI

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Role of Data Science Team Today’s Takeaway Handle Large-scale Data

Slide 3

Slide 3 text

Estimate Business Impacts Role of Data Science Team Suggest Next Improvement Today’s Takeaway Handle Large-scale Data

Slide 4

Slide 4 text

Agenda › Improvement Project of Chat Menu UI › Challenges in Analysis › Further UX Improvement › Wrap-Up

Slide 5

Slide 5 text

Improvement Project of Chat Menu UI

Slide 6

Slide 6 text

Chat Menu Renewal Released in April 2020

Slide 7

Slide 7 text

› Easy to access information in Chatroom › Settings for communication › Management of members › List of shared contents “New” Chat Menu UI

Slide 8

Slide 8 text

Why did we renew Chat Menu?

Slide 9

Slide 9 text

Chat Menu Version 0 › Legacy UI › Many icons and labels › Few clues of what is inside › Limited label length Issues

Slide 10

Slide 10 text

Chat Menu Version 1 Issues › Clicks of album and related: +30% › Clicks of events: -24% › Count of group invite: -29% › Count of friend block and group leave: -20% → Not good in total compared with version 0 (only iOS)

Slide 11

Slide 11 text

Plan for Improvement Displaying Visuals Grouping Items One Screen

Slide 12

Slide 12 text

4 icons Candidate Plans › Take advantages of Version 0 and 1 8 icons Visual Lists & Text Items Visual Lists 4-icon UI 8-icon UI

Slide 13

Slide 13 text

Challenges in Analysis

Slide 14

Slide 14 text

How Can We Decide the Order among 4 UIs? 4-icon UI 8-icon UI Version 0 Version 1

Slide 15

Slide 15 text

Solution: Design 3 Stages of KPIs › Define KPIs in the order of priority › Apply a knockout competition on 3 stages View Click Conversion › View Count › View Days Count › … › Click Count › Click Days Count › Bounce Rate › … › Group Invitation › Note Creation › Friend Block › … First Second Third

Slide 16

Slide 16 text

How Do We Handle All 120 Patterns? 5 Room Types Friend, Group, Chats, OpenChat, Official Account 6 pair 4 Regions JP, TW, TH, ID × × › Need all-pair comparisons? › Require large sample size? 4-icon UI vs. Ver.0 4-icon UI vs. Ver.1 4-con UI vs. 8-icon U 8-icon UI vs. Ver.0 ….

Slide 17

Slide 17 text

Solution: 3-pair Comparisons › Minimize # comparisons pairs based on previous knowledge › Moderate false positive rate with small sample size vs. Ver.0 Ver.1 Comparison 1 Comparison 2 Expectation Why? Ver.0 wins Previous observation 4-icon UI wins Rich visual No idea ? 4-icon UI 8-icon UI vs. vs. Ver.0 4-icon UI Comparison 3

Slide 18

Slide 18 text

Solution: Auto Analysis by Libra Report › Provide semi-automatic data summarization and visualization › Create re-usable tables for test result analysis

Slide 19

Slide 19 text

First Stage: Draw View › View Count → › View Days Count → › … View Count ■ Version 0 ■ Version 1 ■ Version 0 ■ 4-icon UI ■ 4-icon UI ■ 8-icon UI

Slide 20

Slide 20 text

Second Stage: 4-icon UI wins View Click › Click Count +0.01 per user › Click Days Count → › Bounce Rate -9% › … 4-icon UI › View Count → › View Days Count → › …

Slide 21

Slide 21 text

Decision: Choose 4-icon UI View Click Conversion › View Count → › View Days Count → › … › Click Count +0.01 per user › Click Days Count → › Bounce Rate -9% › … › Group Invitation → › Note Creation -0.00006 › Friend Block → …

Slide 22

Slide 22 text

Drastic Changes Does New UI Bias Distort Results? › Increase clicks only when users visit Chat Menu for the first time?

Slide 23

Slide 23 text

Solution: Estimate Effects of New UI Bias 100 110 120 130 140 150 160 Week1 Week2 Week3 Week4 Version 0 4-icon UI › Apply difference-in-difference approach

Slide 24

Slide 24 text

Solution: Estimate Effects of New UI Bias 100 110 120 130 140 150 160 Week1 Week2 Week3 Week4 Version 0 4-icon UI › Apply difference-in-difference approach *Statistically significant decrease

Slide 25

Slide 25 text

How Much Are Business Impacts? × × 166M Click Count: +0.01 Period: 1 year Population among 4 regions › Translate observed effect size into real outcome +0.01 = +600M As of June 2020 click per year

Slide 26

Slide 26 text

Further UX Improvement

Slide 27

Slide 27 text

Is Position of Setting Icon the Best? › Users can click setting icon to change chat room settings

Slide 28

Slide 28 text

Data Analysis Finds Pain Points

Slide 29

Slide 29 text

Data Analysis Finds Pain Points › 40% of sessions: “Link” or “File” → “Setting Icon”

Slide 30

Slide 30 text

Data Analysis Finds Pain Points › +105% Sessions of “Block” → “Unblock” › 40% of sessions: “Link” or “File” → “Setting Icon”

Slide 31

Slide 31 text

Decision: Rearrange Setting Icon

Slide 32

Slide 32 text

› Easy to access information in Chatroom › Settings for communication › Management of members › List of shared contents “New” Chat Menu UI

Slide 33

Slide 33 text

Wrap-Up

Slide 34

Slide 34 text

Role of Data Science Team Handle Large-scale Data Estimate Business Impacts Suggest Next Improvement Today’s Takeaway

Slide 35

Slide 35 text

Thank You