Analytics and UX: Research and design for people who use data

D7424ff0e1dc983d898de62c7496c321?s=47 uxindia
October 25, 2019

Analytics and UX: Research and design for people who use data

UXINDIA2019: India's biggest international conference on User Experience Design which is being celebrated on Sep 9, 10 & 11 at Novotel HICC, Hyderabad.
www.ux-india.org/uxindia2019
#ux #uxindia #uxindia2019 #designthinking #uxi19 #uxconference #umodesign

D7424ff0e1dc983d898de62c7496c321?s=128

uxindia

October 25, 2019
Tweet

Transcript

  1. None
  2. Analytics and UX Understanding and designing for people who use

    data Fadden, S. 2019. Analytics and UX: Understanding and designing for people who use data workshop. UX India, 10 September, Hyderabad.
  3. Steve Fadden, Ph.D. 2 Head of User Research, Measurement &

    Analytics, Google Lecturer, UC Berkeley School of Information @sfadden on Twitter https://www.linkedin.com/in/stevefadden/
  4. Introductions

  5. Who’s here? Experience: <2 years 2-5 years >5 years Role:

    Design Development Research Management Organization: Academia Business Consulting Government Nonprofit
  6. Current situation: Users, Contexts, Systems, Data “User-friendly data takes time,

    effort, and teamwork” Image: https://blogs.nasa.gov/spacestation/author/cewilli3/
  7. Data users ≠ analysts or data scientists

  8. Scenario

  9. Scenario A ridesharing company wants a tool to monitor their

    operational, financial, and customer performance. They would like the tool to provide data to employees so everyone is aware of their metrics, and empowered to make decisions. Image: https://commons.wikimedia.org/wiki/File:India_-_Kolkata_rainy_street_-_3819.jpg
  10. Assignments At each table: • 1-2 “user representatives” • 1-2

    research • 1-2 design • Observer/note-takers A transportation company wants a tool to monitor operational and financial performance. They would like the tool to provide data to employees so everyone is aware of their metrics.
  11. User representative notes You are the Sr. Operations Manager, reporting

    to the office of the CEO/COO. Your group is small, and you are often asked to provide information about operations, service, sales, and sentiment. Your primary duty is the daily monitoring and reporting of errors, uptime, latency for your mobile apps, finance and inventory databases, and customer service software. You’re frustrated that all of this data is in separate systems today. You often need to view and compare your data by region (state, country), date range (weeks, months, quarters), and customer type (luxury, group, economy) When metrics go out of range, you need to take action to investigate the problem, then assign a team, or hire consulting resources, to fix it. Secondary duties depend on time of year, location, staffing shortages, and context - such as if there’s a crisis caused by an accident. These include: • Reporting on driver and passenger sentiment (satisfaction, ratings) • Mobile app engagement, attrition • Current revenue, revenue forecast for next quarter • Media coverage activity
  12. Work ecosystem • Different departments? • Hierarchy (org chart)? •

    Communications? • Greatest need? • Barriers? • Impacts? A transportation company wants a tool to monitor operational and financial performance. They would like the tool to provide data to employees so everyone is aware of their metrics.
  13. Target user • Job role or title • User goals

    • Information needs • Triggers • Actions • Open research questions A transportation company wants a tool to monitor operational and financial performance. They would like the tool to provide data to employees so everyone is aware of their metrics.
  14. Data collection plan

  15. Access to users is essential

  16. Uncovering user needs • Interview • Observation • Survey •

    Diary study • Desk research Image: https://pxhere.com/en/photo/1447775
  17. Interviewing Image: https://pixnio.com/objects/computer/business-businesswoman-laptop-computer-work-office

  18. Interview goals • In-depth understanding ◦ User goals and intentions

    ◦ Needs and frustrations ◦ Techniques and processes ◦ Relationships, dependencies, power ◦ Context of work 17
  19. Typical Structure 18 Set expectations Build trust Draw out the

    story Find useful details Debrief Closure Intro Warm-up Focus Deep focus Retrospective Wrap-up Image: http://pixabay.com/es/reloj-de-arena-reloj-temporizador-152090
  20. Critical incident questions clarify problems • Time since last experience

    • Gather details ◦ Description ◦ Actions taken ◦ Feelings ◦ Outcome ◦ Future actions/responses desired 19 Reference: http://www.usabilitynet.org/tools/criticalincidents.htm
  21. 20 “Consider the last time you needed to use a

    metric. How long ago was this? What metric did you use? Describe the steps you took, and highlight any surprises or problems (if any) that happened. What would you do differently, if you could?” Use critical incident for data needs and usage
  22. Use Grand Tour questions for overviews Guided: “Discuss the most

    important metrics you review.” Task-related: “Talk through the steps you take when you review your weekly metrics.” Typical: “Tell me how you typically respond when this metric goes out of range.” Reference: Larry Wood, 1997. Semi-structured interviewing for user-centered design, Interactions.
  23. Use Talkthroughs to understand processes Concurrent think-aloud: “Talk through your

    thoughts as you consider each of these metrics.” Aided recall: “Look through this spreadsheet and talk about the thoughts you had, and the actions you took.” Cross-examination: “How did you decide on taking this action vs. a different one?” Reference: Larry Wood, 1997. Semi-structured interviewing for user-centered design, Interactions.
  24. Identify needs and questions users (unknowingly) ask of their data

  25. Objective: Uncover data needs and questions Who: Business manager What:

    Needs revenue data Why: To understand business health To have answers for Leadership When: Daily, in morning Where: Anywhere I happen to be How: By Product, Timeline (day, week, month, quarter, year) Next: Check engagement metrics Consult with peers Develop action plans Question: “What is our current revenue? Is it on target?”
  26. Create question flow 1. Compile questions by goal 2. Order

    questions 3. Identify metrics needed 4. Assign priorities to questions Question flow Question1 Is revenue on target? Answer1a [done] (Yes: $5M) Answer1b (No: $3M, $2M below) Question1.1 Which products are not selling? Answer1.1a (ABC, DEF, XYZ) Question1.2 Which regions are not selling? Answer1.2a (State1) Question1.3 Which segments are not selling? Answer1.3a (Small business) Question1.1.1 How can I best close the gap? Answer1.1.1a (Discount ABC by 10%) Answer1.1.1b (Increase sales team in State1) Question2 Is service on target? Answer2a [done] (Yes, 100% of P1 tickets resolved in <72 hours) Answer2b (No, 25% of P1 tickets resolved in >72 hours) Question2.1 Which products are associated with slow service? Answer2.1a (XYZ) Question 2.2 Which regions are associated with slow service? Answer2.2a (State1, State2) Answer2.2b Which segments are associated with slow service? Answer2.2c (Small business) Reference: https://medium.com/salesforce-ux/transforming-data-to-insights-773d25acd53f
  27. Assign metrics and priorities • Use goals and tasks to

    identify metrics ◦ Work with users ◦ Look at context of use • Create metric definitions • Prioritize metrics importance • Example: Is revenue on target? a. Revenue: Total amount of money earned b. Target definition: % revenue increase by quarter c. Priority: Highest (P1) • Example: Is service on target? a. Service: Total number of P1 tickets resolved b. Target definition: Resolution within 72 hours c. Priority: Second-highest (P2) Reference: https://medium.com/salesforce-ux/transforming-data-to-insights-773d25acd53f
  28. Practice

  29. 28 Individual: Develop 3 interview questions 1. Critical incident 2.

    Grand tour a. Guided b. Task-related c. Typical 3. Talkthroughs a. Concurrent think-aloud b. Aided recall Cross-examination “Consider the last time you needed to use a metric. How long ago was this? What metric did you use? Describe the steps you took, and highlight any surprises or problems (if any) that happened. What would you do differently, if you could?” “Discuss the most important metrics you review.” “Talk through the steps you take when you review your weekly metrics.” “Tell me how you typically respond when this metric goes out of range.” “Talk through your thoughts as you consider each of these metrics.” “Look through this spreadsheet and talk about the thoughts you had, and the actions you took.” “How did you decide on taking this action vs. a different one?”
  30. 29 Table: Create interview script 1. Critical incident 2. Grand

    tour a. Guided b. Task-related c. Typical 3. Talkthroughs a. Concurrent think-aloud b. Aided recall c. Cross-examination • Consider follow-ups to clarify needs and goals ◦ “Why do you need this number?” ◦ “What other numbers do you need?” ◦ “How do you use it?” ◦ “When/where do you need it?” ◦ “What do you do after you know this number?”
  31. Table: Interview user representative(s) • Researcher(s): Conduct interview with your

    user representative(s) • Note-taker(s): Capture notes and observations • Everyone else: Listen, propose follow-up questions 1. Follow script 2. Ask 1 of each: ◦ Critical incident ◦ Grand tour ◦ Talkthrough 3. Identify ◦ Overall goals ◦ User’s data questions ◦ Order of questions ◦ Relative priority of each question
  32. Report out • Challenges • Highlights • Changes • Insights

    31
  33. Data feasibility and availability

  34. Consider current reality as well as future possibility

  35. Hierarchy of data needs Assess the state of your data

    1. Available & clean? ◦ Data Engineering 2. Calculated & combined? ◦ Business Intelligence/Analytics 3. Algorithms & models applied? ◦ Data Science, Machine Learning ? Prescription Prediction Description Collection https://blog.treasuredata.com/blog/2016/03/17/the-analytics-hierarchy-of-needs/
  36. What’s possible today vs. future? 1. Identify data readiness for

    goal-related data a. Available now b. Near-term investments c. Longer-term investments 2. Explore opportunities to develop new data a. Predictions, prescriptions, descriptions b. Collection needs ? Prescription Prediction Description Collection https://blog.treasuredata.com/blog/2016/03/17/the-analytics-hierarchy-of-needs/
  37. The data story

  38. What’s the purpose of the data display? 3 dashboard categories

    • Operations: Answer questions ◦ Top down: High level indicators • Analytics: Explore data ◦ Bottom up: Granular details • Presentation: Curated snapshot ◦ KPI: Monitor important metrics Image: https://informationisbeautiful.net/visualizations/what-makes-a-good-data-visualization/
  39. What kind of data story is needed? Consider story structure,

    based on: • Persona? • Goals? • Contexts? • Priorities? Consider story flow: • 1s story • 10s story • 1 minute story • 10 minute story Reference: https://medium.com/salesforce-ux/transforming-data-to-insights-773d25acd53f
  40. Which patterns make sense? Typical elements • Alerts • “To

    do” items • Performance statistics • Current status • Search • Task starting points • Social components • Recent activity • News, events, announcements • Push content Reference: https://www.designforcontext.com/insights/designing-great-dashboards-saas-and-enterprise-applications, Image: https://medium.com/@yifei.liu/https-medium-com-yifei-liu-lets-talk-about-dashboard-design-c63cd1a45b3f
  41. Data presentation

  42. Presentation should support your 1s, 10s, 1m, 10m story goals

  43. Focus and visual hierarchy Influenced by: • Size • Color

    • Contrast • Alignment • Repetition • Proximity • Whitespace • Texture and Style Primary Secondary Tertiary Reference: https://www.topcoder.com/blog/10-useful-design-techniques-master-visual-hierarchy/
  44. Scan patterns F Z Layer cake References: https://instapage.com/blog/z-pattern-layout, https://99designs.com/blog/tips/visual-hierarchy-landing-page-designs/, https://www.nngroup.com/articles/layer-cake-pattern-scanning/

  45. Dashboards • Operations: Monitoring - answer questions • Analytics: Exploration

    - discover insights • Presentation: Summarize - provide overview Reference: https://material.io/design/communication/data-visualization.html#dashboards
  46. Dashboards • Operations: Monitoring - answer questions • Analytics: Exploration

    - discover insights • Presentation: Summarize - provide overview Reference: https://material.io/design/communication/data-visualization.html#dashboards
  47. Dashboards • Operations: Monitoring - answer questions • Analytics: Exploration

    - discover insights • Presentation: Summarize - provide overview Reference: https://material.io/design/communication/data-visualization.html#dashboards
  48. Example layouts Summary Content Actions Details Filters Summary Content Details

    / Actions Filters Summary Content Actions Filters Content Content Summary Details Reference: https://medium.com/salesforce-ux/transforming-data-to-insights-773d25acd53f
  49. Example layout: Operations When did the issue occur? Where did

    the issue occur? What else is affected? What issues need my attention? Reference: https://material.io/design/communication/data-visualization.html#dashboards
  50. Visualizations should be comprehensible with little/no training

  51. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts, Image: https://stats.wikimedia.org/v2/#/all-projects
  52. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts, Image: https://stats.wikimedia.org/v2/#/all-projects/contributing/top-edited-pages/normal|table|last-month|~total|monthly
  53. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts
  54. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts
  55. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts, Image: https://stats.wikimedia.org/v2/#/all-projects/reading/total-page-views/normal|bar|2-year|~total|monthly
  56. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts, Image: https://material.io/design/communication/data-visualization.html#style
  57. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts, Image: https://material.io/design/communication/data-visualization.html#style
  58. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts, Image: https://stats.wikimedia.org/v2/#/all-projects/contributing/top-edited-pages/normal|table|last-month|~total|monthly
  59. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts
  60. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts, Image: https://stats.wikimedia.org/v2/#/en.wikipedia.org/reading/page-views-by-country/normal|map|last-month|~total|monthly
  61. Visualization • Numbers • Bar/column • Line • Area •

    Other Reference: https://material.io/design/communication/data-visualization.html#selecting-charts, Image: https://material.io/design/communication/data-visualization.html#style
  62. Visualization Types • Change over time • Comparison • Ranking

    • Part-to-whole • Correlation • Distribution • Flow • Relationship Reference: https://material.io/design/communication/data-visualization.html#types
  63. Visualization Types • Change over time • Comparison • Ranking

    • Part-to-whole • Correlation • Distribution • Flow • Relationship Reference: https://material.io/design/communication/data-visualization.html#types
  64. Visualization Types • Change over time • Comparison • Ranking

    • Part-to-whole • Correlation • Distribution • Flow • Relationship Reference: https://material.io/design/communication/data-visualization.html#types
  65. Visualization Types • Change over time • Comparison • Ranking

    • Part-to-whole • Correlation • Distribution • Flow • Relationship Reference: https://material.io/design/communication/data-visualization.html#types
  66. Visualization Types • Change over time • Comparison • Ranking

    • Part-to-whole • Correlation • Distribution • Flow • Relationship Reference: https://material.io/design/communication/data-visualization.html#types
  67. Visualization Types • Change over time • Comparison • Ranking

    • Part-to-whole • Correlation • Distribution • Flow • Relationship Reference: https://material.io/design/communication/data-visualization.html#types
  68. Visualization Types • Change over time • Comparison • Ranking

    • Part-to-whole • Correlation • Distribution • Flow • Relationship Reference: https://material.io/design/communication/data-visualization.html#types
  69. Visualization Types • Change over time • Comparison • Ranking

    • Part-to-whole • Correlation • Distribution • Flow • Relationship Reference: https://material.io/design/communication/data-visualization.html#types
  70. Group activity Determine purpose and story • What do your

    users need? • What is their context? Consider your approach • How to meet user needs • What data visualizations and why? • Usage scenario: How to support 1s, 10s, 1m, 10m? Each table 1. Review user goals, question flow 2. Ideate (crazy-8s?) and share 3. Discuss: Key visualizations to use 4. Discuss: How to support 1s, 10s, 1m, 10m 5. Write the scenario: Describe how your user will use this tool
  71. Report out • Process ◦ User needs and contexts? ◦

    Approach? ◦ Story ideas? 70
  72. Assessment

  73. Evaluate your visualizations, story, and presentation flow for comprehensibility

  74. Review visualizations • Assess content • 24 guidelines ◦ Text

    ◦ Arrangement ◦ Color ◦ Lines ◦ Overall Reference: https://stephanieevergreen.com/interactive-data-visualization-checklist/
  75. Evaluate usability Evaluation • 5 participants (per user profile) •

    Ensure users can complete realistic tasks with no guidance • Use high-fidelity prototype or production tool • Think aloud protocol Image: https://www.flickr.com/photos/eekim/1819104307
  76. Gather feedback Embedded survey • 3 questions: ◦ Goal of

    visit ◦ Ease of use rating ◦ Reason for rating • Monitor over time • Review concerns Image: https://www.flickr.com/photos/64763706@N08/6850650385
  77. Monitor funnel Process: How a user is exposed to tool

    and ultimately engages with it 1. Awareness 2. Interest 3. Desire 4. Action / Conversion 5. Re-engagement Reference: https://www.bigcommerce.com/blog/conversion-rate-optimization-conversion-funnel/#what-is-a-conversion-funnel, Image: https://pixabay.com/vectors/infographic-funnel-chart-marketing-2944842/
  78. Funnel approach Establish metrics and ratios • Awareness:Desire: 50% visitors

    sign up for training or account • Conversions: 90% Leadership over 6 months; 50% employees over 6 months • Re-engagement: 90% Monthly Active Users Examples • Awareness: First-time site visits; Email opens • Interest: Repeat visits; Intro meeting attendance • Desire: Training sign-up; Account request • Conversion: Logins; Tool use • Re-engagement: Repeated tool use (daily, weekly, monthly) Reference: https://www.bigcommerce.com/blog/conversion-rate-optimization-conversion-funnel/#what-is-a-conversion-funnel
  79. Consider logic model Relationship between program resources, activities, and outcomes

    • Resources (inputs) • Activities • Outputs • Outcomes (attitude, behavior, knowledge, skills, status) • Impact (system change) Reference: http://toolkit.pellinstitute.org/evaluation-guide/plan-budget/using-a-logic-model/, Image: https://www.nicepik.com/person-drawing-flowchart-mark-marker-hand-leave-production-planning-control-organizational-structure-free-photo-959325
  80. Performance metrics tool for decision-making Resources Activities Outputs Outcomes (awareness)

    Leadership Management Analysts Line personnel Data engineering costs Training material costs Leadership training Employee training Data reviews and assessments Tool modification and extension # leaders trained # employees trained # hours committed # tool updates % leadership aware of tool (poll) % employees aware of tool (poll) # of meetings where tool is discussed # of requests for training Outcomes (behavior) % leadership decisions made with tool % employee decisions made with tool # tool feature requests # times tool used (daily, weekly, monthly) Impact Better decision- making by leadership Better decision- making by employees Greater employee cohesion, buy-in
  81. Closing considerations

  82. Clarify terminology Image: https://www.pexels.com/photo/black-and-white-book-business-close-up-267669/

  83. Support collaboration and awareness Image: https://pxhere.com/en/photo/1549037

  84. Anticipate growth with patterns and templates Image: https://pixabay.com/photos/book-books-library-literature-4007822/

  85. Provide recommended actions https://www.flickr.com/photos/howardlake/4141454994

  86. Consider anomaly detection Image: https://www.maxpixel.net/Expand-Geography-Magnifying-Glass-Map-Discover-1277578

  87. Raise organizational data awareness https://www.flickr.com/photos/neychurluvr/3448529469

  88. Further reading Understanding and displaying data: Evergreen, S. (2017). Effective

    Data Visualization: The Right Chart for the Right Data. Thousand Oaks, CA: SAGE Publications. Few, S. (2004). Show Me the Numbers: Designing Tables and Graphs to Enlighten. Oakland, CA: Analytics Press. Few, S. (2006). Information Dashboard Design: The Effective Visual Communication of Data, Sebastopol, CA: O’Reilly Media. McCandless, D. (2012). Information is Beautiful. London, UK: HarperCollins Publishers. Nussbaumer Knaflic, C. (2015). Storytelling with Data: A Data Visualization Guide for Business Professionals. Hoboken, NJ: Tufte, E. (2001). The Visual Display of Quantitative Information. Cheshire, CT: Graphics Press. Wong, D.M. (2010). The Wall Street Journal Guide to Information Graphics: The Dos and Don'ts of Presenting Data, Facts, and Figures. New York, NY: W.W. Norton & Company. Understanding user needs: Hackos, J.T., and Redish, J.C. (1998). User and Task Analysis for Interface Design. New York, NY: John Wiley & Sons. Portigal, S. (2013). Interviewing Users: How to Uncover Compelling Insights. Brooklyn, NY: Rosenfeld Media. John Wiley & Sons.
  89. Questions?

  90. Thank you!