Save 37% off PRO during our Black Friday Sale! »

Using diary studies to track long-term usability results

Using diary studies to track long-term usability results

Presented to PSEWEB 2018. Overview and considerations for conducting diary studies in higher education.


Amy Grace Wells

August 14, 2018


  1. Using diary studies to track long-term usability results Amy Grace

    Wells, UX Booth @amygracewells
  2. Hi, I’m Amy Grace and I’m a crazy person @amygracewells

  3. None
  4. Why research? @amygracewells #pseweb

  5. You≠User @amygracewells #pseweb

  6. Prioritize resources @amygracewells #pseweb

  7. Reduce problems @amygracewells #pseweb

  8. What is this diary study thing you speak of? @amygracewells

  9. –Nielsen Norman Group “A diary study is a research method

    used to collect qualitative data about user behaviors, activities, and experiences over time.”
  10. Key attributes • Qualitative • Self-reported at regular intervals •

    Contextual understanding of user behaviors • Good for “how” questions @amygracewells #pseweb
  11. Understand behavior • Habits — time of day; how they

    choose to share • Usage scenarios — primary tasks; workflows for completing longer- term tasks (test these later in the process) • Attitudes and motivations — motivation to perform specific tasks; users feeling and thinking • Changes in behaviors and perceptions — learnable; loyalty; perception • Customer journeys — cross-channel user experience across different devices and channels; cumulative effect of multiple touchpoints @amygracewells #pseweb
  12. Why diary studies? @amygracewells #pseweb

  13. The “poor man’s field study” aka perfectly designed for higher

    ed @amygracewells #pseweb
  14. Learnability & Performance @amygracewells #pseweb

  15. Pros • Immediate reaction, not remembering • Test systems that

    are used regularly • Effect of environment • More in-depth than a single usability study @amygracewells #pseweb
  16. Cons • No observation • Complicated and time-intensive to set

    up • Retaining participants can be difficult • Qualitative results take more time to interpret @amygracewells #pseweb
  17. When to use diary studies? @amygracewells #pseweb

  18. Diary studies to • Track long processes over several days

    (finding and registering for classes, reserving campus space) • Discover what motivates users to initiate certain actions (using search bar) • Determine how a system factors into regular habits • Assess retention and/or learnability @amygracewells #pseweb
  19. How to do a diary study @amygracewells #pseweb

  20. Include • Clear instructions • Template • Collection method •

    Rewards/Encouragement @amygracewells #pseweb
  21. Disclaimer Know your university rules (or don’t). Some may require

    approval of research offices/IRB. @amygracewells
  22. Instructions • Be explicit • When/how often should they report

    • How they should report • What they should report • Provide help • Direct contact info @amygracewells #pseweb
  23. None
  24. Template • Gets the info you want with a question-based

    template • Makes it easier for the participant • Allows for limited quantitative questions @amygracewells #pseweb
  25. None
  26. Allow space for activities, thoughts, and frustrations @amygracewells #pseweb

  27. Collection method • Email • Evernote • SMS • SurveyMonkey

    • Google Drive (doc or form) • UserTesting • Voicemails @amygracewells #pseweb
  28. Collection consideration • Audience • Environment • Comfort with technology

    • Allow for media @amygracewells #pseweb
  29. Retention • Choose rewards that fit your budget (gift cards,

    swag) • Set milestones for rewards and do your part • Send reminders and thanks @amygracewells #pseweb
  30. Don’t miss post-study opportunities Conduct moderated interviews to discuss overall

    experience. Clarify any details or ask any questions from the results. @amygracewells #pseweb
  31. None
  32. How do we recruit? @amygracewells #pseweb

  33. Choose wisely • Goal is 4-6 participants • Choose 10-12

    from target audience • Conduct interviews (bonus if you can provide incentive for this) • Select those that answer freely and most likely to have time to participate (aka don’t choose an athlete during peak season) @amygracewells #pseweb
  34. Set up success • Conduct a training • Provide feedback

    after the first submission (but keep it minimal; you’re not here to bias) • Thank them after submission • Remind about upcoming reward milestones @amygracewells #pseweb
  35. How do we use results? @amygracewells #pseweb

  36. None
  37. Qualitative results require time, time, & more time Look for

    patterns Color-code Save repeated words and phrases @amygracewells #pseweb
  38. Map their experience Grab the post-its Map their key tasks

    Map their obstacles Map their emotions @amygracewells #pseweb
  39. Let their words make the impact Include quotes in results

    report @amygracewells #pseweb
  40. Results inform • Future testing by identifying issues • Iterative

    testing such as usability studies • Resource allocation • Case for more testing and testing budgets @amygracewells #pseweb
  41. Thanks for listening Amy Grace Wells Editorial Director, UX Booth @amygracewells
  42. Sources • Nielsen Norman Group • User Testing • UXMastery

    • UX Booth • Images from: 123rf, Pinterest, BlinkUX, N/N Group, Spotless