Using diary studies to track long-term usability results

Presented to PSEWEB 2018. Overview and considerations for conducting diary studies in higher education.

1. Using diary studies to track long-term usability results Amy Grace Wells, UX Booth @amygracewells
2. Hi, I’m Amy Grace and I’m a crazy person @amygracewells #pseweb
4. Why research? @amygracewells #pseweb
5. You≠User @amygracewells #pseweb
6. Prioritize resources @amygracewells #pseweb
7. Reduce problems @amygracewells #pseweb
8. What is this diary study thing you speak of? @amygracewells #pseweb
9. –Nielsen Norman Group “A diary study is a research method used to collect qualitative data about user behaviors, activities, and experiences over time.”
10. Key attributes • Qualitative • Self-reported at regular intervals • Contextual understanding of user behaviors • Good for “how” questions @amygracewells #pseweb
11. Understand behavior • Habits — time of day; how they choose to share • Usage scenarios — primary tasks; workflows for completing longer- term tasks (test these later in the process) • Attitudes and motivations — motivation to perform specific tasks; users feeling and thinking • Changes in behaviors and perceptions — learnable; loyalty; perception • Customer journeys — cross-channel user experience across different devices and channels; cumulative effect of multiple touchpoints @amygracewells #pseweb
12. Why diary studies? @amygracewells #pseweb
13. The “poor man’s field study” aka perfectly designed for higher ed @amygracewells #pseweb
14. Learnability & Performance @amygracewells #pseweb
15. Pros • Immediate reaction, not remembering • Test systems that are used regularly • Effect of environment • More in-depth than a single usability study @amygracewells #pseweb
16. Cons • No observation • Complicated and time-intensive to set up • Retaining participants can be difficult • Qualitative results take more time to interpret @amygracewells #pseweb
17. When to use diary studies? @amygracewells #pseweb
18. Diary studies to • Track long processes over several days (finding and registering for classes, reserving campus space) • Discover what motivates users to initiate certain actions (using search bar) • Determine how a system factors into regular habits • Assess retention and/or learnability @amygracewells #pseweb
19. How to do a diary study @amygracewells #pseweb
20. Include • Clear instructions • Template • Collection method • Rewards/Encouragement @amygracewells #pseweb
21. Disclaimer Know your university rules (or don’t). Some may require approval of research offices/IRB. @amygracewells
22. Instructions • Be explicit • When/how often should they report • How they should report • What they should report • Provide help • Direct contact info @amygracewells #pseweb
24. Template • Gets the info you want with a question-based template • Makes it easier for the participant • Allows for limited quantitative questions @amygracewells #pseweb
26. Allow space for activities, thoughts, and frustrations @amygracewells #pseweb
27. Collection method • Email • Evernote • SMS • SurveyMonkey • Google Drive (doc or form) • UserTesting • Voicemails @amygracewells #pseweb
28. Collection consideration • Audience • Environment • Comfort with technology • Allow for media @amygracewells #pseweb
29. Retention • Choose rewards that fit your budget (gift cards, swag) • Set milestones for rewards and do your part • Send reminders and thanks @amygracewells #pseweb
30. Don’t miss post-study opportunities Conduct moderated interviews to discuss overall experience. Clarify any details or ask any questions from the results. @amygracewells #pseweb
32. How do we recruit? @amygracewells #pseweb
33. Choose wisely • Goal is 4-6 participants • Choose 10-12 from target audience • Conduct interviews (bonus if you can provide incentive for this) • Select those that answer freely and most likely to have time to participate (aka don’t choose an athlete during peak season) @amygracewells #pseweb
34. Set up success • Conduct a training • Provide feedback after the first submission (but keep it minimal; you’re not here to bias) • Thank them after submission • Remind about upcoming reward milestones @amygracewells #pseweb
35. How do we use results? @amygracewells #pseweb
37. Qualitative results require time, time, & more time Look for patterns Color-code Save repeated words and phrases @amygracewells #pseweb
38. Map their experience Grab the post-its Map their key tasks Map their obstacles Map their emotions @amygracewells #pseweb
39. Let their words make the impact Include quotes in results report @amygracewells #pseweb
40. Results inform • Future testing by identifying issues • Iterative testing such as usability studies • Resource allocation • Case for more testing and testing budgets @amygracewells #pseweb
41. Thanks for listening Amy Grace Wells Editorial Director, UX Booth amygrace@uxbooth.com @amygracewells
42. Sources • Nielsen Norman Group • User Testing • UXMastery • UX Booth • Images from: 123rf, Pinterest, BlinkUX, N/N Group, Spotless