Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Science leads overhaul of the chat menu UI

Data Science leads overhaul of the chat menu UI

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 27, 2020
Tweet

More Decks by LINE DevDay 2020

Other Decks in Technology

Transcript

  1. None
  2. Role of Data Science Team Today’s Takeaway Handle Large-scale Data

  3. Estimate Business Impacts Role of Data Science Team Suggest Next

    Improvement Today’s Takeaway Handle Large-scale Data
  4. Agenda › Improvement Project of Chat Menu UI › Challenges

    in Analysis › Further UX Improvement › Wrap-Up
  5. Improvement Project of Chat Menu UI

  6. Chat Menu Renewal Released in April 2020

  7. › Easy to access information in Chatroom › Settings for

    communication › Management of members › List of shared contents “New” Chat Menu UI
  8. Why did we renew Chat Menu?

  9. Chat Menu Version 0 › Legacy UI › Many icons

    and labels › Few clues of what is inside › Limited label length Issues
  10. Chat Menu Version 1 Issues › Clicks of album and

    related: +30% › Clicks of events: -24% › Count of group invite: -29% › Count of friend block and group leave: -20% → Not good in total compared with version 0 (only iOS)
  11. Plan for Improvement Displaying Visuals Grouping Items One Screen

  12. 4 icons Candidate Plans › Take advantages of Version 0

    and 1 8 icons Visual Lists & Text Items Visual Lists 4-icon UI 8-icon UI
  13. Challenges in Analysis

  14. How Can We Decide the Order among 4 UIs? 4-icon

    UI 8-icon UI Version 0 Version 1
  15. Solution: Design 3 Stages of KPIs › Define KPIs in

    the order of priority › Apply a knockout competition on 3 stages View Click Conversion › View Count › View Days Count › … › Click Count › Click Days Count › Bounce Rate › … › Group Invitation › Note Creation › Friend Block › … First Second Third
  16. How Do We Handle All 120 Patterns? 5 Room Types

    Friend, Group, Chats, OpenChat, Official Account 6 pair 4 Regions JP, TW, TH, ID × × › Need all-pair comparisons? › Require large sample size? 4-icon UI vs. Ver.0 4-icon UI vs. Ver.1 4-con UI vs. 8-icon U 8-icon UI vs. Ver.0 ….
  17. Solution: 3-pair Comparisons › Minimize # comparisons pairs based on

    previous knowledge › Moderate false positive rate with small sample size vs. Ver.0 Ver.1 Comparison 1 Comparison 2 Expectation Why? Ver.0 wins Previous observation 4-icon UI wins Rich visual No idea ? 4-icon UI 8-icon UI vs. vs. Ver.0 4-icon UI Comparison 3
  18. Solution: Auto Analysis by Libra Report › Provide semi-automatic data

    summarization and visualization › Create re-usable tables for test result analysis
  19. First Stage: Draw View › View Count → › View

    Days Count → › … View Count ▪ Version 0 ▪ Version 1 ▪ Version 0 ▪ 4-icon UI ▪ 4-icon UI ▪ 8-icon UI
  20. Second Stage: 4-icon UI wins View Click › Click Count

    +0.01 per user › Click Days Count → › Bounce Rate -9% › … 4-icon UI › View Count → › View Days Count → › …
  21. Decision: Choose 4-icon UI View Click Conversion › View Count

    → › View Days Count → › … › Click Count +0.01 per user › Click Days Count → › Bounce Rate -9% › … › Group Invitation → › Note Creation -0.00006 › Friend Block → …
  22. Drastic Changes Does New UI Bias Distort Results? › Increase

    clicks only when users visit Chat Menu for the first time?
  23. Solution: Estimate Effects of New UI Bias 100 110 120

    130 140 150 160 Week1 Week2 Week3 Week4 Version 0 4-icon UI › Apply difference-in-difference approach
  24. Solution: Estimate Effects of New UI Bias 100 110 120

    130 140 150 160 Week1 Week2 Week3 Week4 Version 0 4-icon UI › Apply difference-in-difference approach *Statistically significant decrease
  25. How Much Are Business Impacts? × × 166M Click Count:

    +0.01 Period: 1 year Population among 4 regions › Translate observed effect size into real outcome +0.01 = +600M As of June 2020 click per year
  26. Further UX Improvement

  27. Is Position of Setting Icon the Best? › Users can

    click setting icon to change chat room settings
  28. Data Analysis Finds Pain Points

  29. Data Analysis Finds Pain Points › 40% of sessions: “Link”

    or “File” → “Setting Icon”
  30. Data Analysis Finds Pain Points › +105% Sessions of “Block”

    → “Unblock” › 40% of sessions: “Link” or “File” → “Setting Icon”
  31. Decision: Rearrange Setting Icon

  32. › Easy to access information in Chatroom › Settings for

    communication › Management of members › List of shared contents “New” Chat Menu UI
  33. Wrap-Up

  34. Role of Data Science Team Handle Large-scale Data Estimate Business

    Impacts Suggest Next Improvement Today’s Takeaway
  35. Thank You