Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using diary studies to track long-term usability results

Using diary studies to track long-term usability results

Presented to PSEWEB 2018. Overview and considerations for conducting diary studies in higher education.

Amy Grace Wells

August 14, 2018

More Decks by Amy Grace Wells

Other Decks in Education


  1. –Nielsen Norman Group “A diary study is a research method

    used to collect qualitative data about user behaviors, activities, and experiences over time.”
  2. Key attributes • Qualitative • Self-reported at regular intervals •

    Contextual understanding of user behaviors • Good for “how” questions @amygracewells #pseweb
  3. Understand behavior • Habits — time of day; how they

    choose to share • Usage scenarios — primary tasks; workflows for completing longer- term tasks (test these later in the process) • Attitudes and motivations — motivation to perform specific tasks; users feeling and thinking • Changes in behaviors and perceptions — learnable; loyalty; perception • Customer journeys — cross-channel user experience across different devices and channels; cumulative effect of multiple touchpoints @amygracewells #pseweb
  4. Pros • Immediate reaction, not remembering • Test systems that

    are used regularly • Effect of environment • More in-depth than a single usability study @amygracewells #pseweb
  5. Cons • No observation • Complicated and time-intensive to set

    up • Retaining participants can be difficult • Qualitative results take more time to interpret @amygracewells #pseweb
  6. Diary studies to • Track long processes over several days

    (finding and registering for classes, reserving campus space) • Discover what motivates users to initiate certain actions (using search bar) • Determine how a system factors into regular habits • Assess retention and/or learnability @amygracewells #pseweb
  7. Include • Clear instructions • Template • Collection method •

    Rewards/Encouragement @amygracewells #pseweb
  8. Disclaimer Know your university rules (or don’t). Some may require

    approval of research offices/IRB. @amygracewells
  9. Instructions • Be explicit • When/how often should they report

    • How they should report • What they should report • Provide help • Direct contact info @amygracewells #pseweb
  10. Template • Gets the info you want with a question-based

    template • Makes it easier for the participant • Allows for limited quantitative questions @amygracewells #pseweb
  11. Collection method • Email • Evernote • SMS • SurveyMonkey

    • Google Drive (doc or form) • UserTesting • Voicemails @amygracewells #pseweb
  12. Retention • Choose rewards that fit your budget (gift cards,

    swag) • Set milestones for rewards and do your part • Send reminders and thanks @amygracewells #pseweb
  13. Don’t miss post-study opportunities Conduct moderated interviews to discuss overall

    experience. Clarify any details or ask any questions from the results. @amygracewells #pseweb
  14. Choose wisely • Goal is 4-6 participants • Choose 10-12

    from target audience • Conduct interviews (bonus if you can provide incentive for this) • Select those that answer freely and most likely to have time to participate (aka don’t choose an athlete during peak season) @amygracewells #pseweb
  15. Set up success • Conduct a training • Provide feedback

    after the first submission (but keep it minimal; you’re not here to bias) • Thank them after submission • Remind about upcoming reward milestones @amygracewells #pseweb
  16. Qualitative results require time, time, & more time Look for

    patterns Color-code Save repeated words and phrases @amygracewells #pseweb
  17. Map their experience Grab the post-its Map their key tasks

    Map their obstacles Map their emotions @amygracewells #pseweb
  18. Results inform • Future testing by identifying issues • Iterative

    testing such as usability studies • Resource allocation • Case for more testing and testing budgets @amygracewells #pseweb
  19. Sources • Nielsen Norman Group • User Testing • UXMastery

    • UX Booth • Images from: 123rf, Pinterest, BlinkUX, N/N Group, Spotless