Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Science leads overhaul of the chat menu UI

Data Science leads overhaul of the chat menu UI

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 27, 2020
Tweet

Transcript

  1. None
  2. Role of Data Science Team Today’s Takeaway Handle Large-scale Data

  3. Estimate Business Impacts Role of Data Science Team Suggest Next

    Improvement Today’s Takeaway Handle Large-scale Data
  4. Agenda › Improvement Project of Chat Menu UI › Challenges

    in Analysis › Further UX Improvement › Wrap-Up
  5. Improvement Project of Chat Menu UI

  6. Chat Menu Renewal Released in April 2020

  7. › Easy to access information in Chatroom › Settings for

    communication › Management of members › List of shared contents “New” Chat Menu UI
  8. Why did we renew Chat Menu?

  9. Chat Menu Version 0 › Legacy UI › Many icons

    and labels › Few clues of what is inside › Limited label length Issues
  10. Chat Menu Version 1 Issues › Clicks of album and

    related: +30% › Clicks of events: -24% › Count of group invite: -29% › Count of friend block and group leave: -20% → Not good in total compared with version 0 (only iOS)
  11. Plan for Improvement Displaying Visuals Grouping Items One Screen

  12. 4 icons Candidate Plans › Take advantages of Version 0

    and 1 8 icons Visual Lists & Text Items Visual Lists 4-icon UI 8-icon UI
  13. Challenges in Analysis

  14. How Can We Decide the Order among 4 UIs? 4-icon

    UI 8-icon UI Version 0 Version 1
  15. Solution: Design 3 Stages of KPIs › Define KPIs in

    the order of priority › Apply a knockout competition on 3 stages View Click Conversion › View Count › View Days Count › … › Click Count › Click Days Count › Bounce Rate › … › Group Invitation › Note Creation › Friend Block › … First Second Third
  16. How Do We Handle All 120 Patterns? 5 Room Types

    Friend, Group, Chats, OpenChat, Official Account 6 pair 4 Regions JP, TW, TH, ID × × › Need all-pair comparisons? › Require large sample size? 4-icon UI vs. Ver.0 4-icon UI vs. Ver.1 4-con UI vs. 8-icon U 8-icon UI vs. Ver.0 ….
  17. Solution: 3-pair Comparisons › Minimize # comparisons pairs based on

    previous knowledge › Moderate false positive rate with small sample size vs. Ver.0 Ver.1 Comparison 1 Comparison 2 Expectation Why? Ver.0 wins Previous observation 4-icon UI wins Rich visual No idea ? 4-icon UI 8-icon UI vs. vs. Ver.0 4-icon UI Comparison 3
  18. Solution: Auto Analysis by Libra Report › Provide semi-automatic data

    summarization and visualization › Create re-usable tables for test result analysis
  19. First Stage: Draw View › View Count → › View

    Days Count → › … View Count ▪ Version 0 ▪ Version 1 ▪ Version 0 ▪ 4-icon UI ▪ 4-icon UI ▪ 8-icon UI
  20. Second Stage: 4-icon UI wins View Click › Click Count

    +0.01 per user › Click Days Count → › Bounce Rate -9% › … 4-icon UI › View Count → › View Days Count → › …
  21. Decision: Choose 4-icon UI View Click Conversion › View Count

    → › View Days Count → › … › Click Count +0.01 per user › Click Days Count → › Bounce Rate -9% › … › Group Invitation → › Note Creation -0.00006 › Friend Block → …
  22. Drastic Changes Does New UI Bias Distort Results? › Increase

    clicks only when users visit Chat Menu for the first time?
  23. Solution: Estimate Effects of New UI Bias 100 110 120

    130 140 150 160 Week1 Week2 Week3 Week4 Version 0 4-icon UI › Apply difference-in-difference approach
  24. Solution: Estimate Effects of New UI Bias 100 110 120

    130 140 150 160 Week1 Week2 Week3 Week4 Version 0 4-icon UI › Apply difference-in-difference approach *Statistically significant decrease
  25. How Much Are Business Impacts? × × 166M Click Count:

    +0.01 Period: 1 year Population among 4 regions › Translate observed effect size into real outcome +0.01 = +600M As of June 2020 click per year
  26. Further UX Improvement

  27. Is Position of Setting Icon the Best? › Users can

    click setting icon to change chat room settings
  28. Data Analysis Finds Pain Points

  29. Data Analysis Finds Pain Points › 40% of sessions: “Link”

    or “File” → “Setting Icon”
  30. Data Analysis Finds Pain Points › +105% Sessions of “Block”

    → “Unblock” › 40% of sessions: “Link” or “File” → “Setting Icon”
  31. Decision: Rearrange Setting Icon

  32. › Easy to access information in Chatroom › Settings for

    communication › Management of members › List of shared contents “New” Chat Menu UI
  33. Wrap-Up

  34. Role of Data Science Team Handle Large-scale Data Estimate Business

    Impacts Suggest Next Improvement Today’s Takeaway
  35. Thank You