Slide 1
Slide 1 text
Using diary studies
to track long-term
usability results
Amy Grace Wells, UX Booth
@amygracewells
Slide 2
Slide 2 text
Hi, I’m Amy Grace
and I’m a crazy person
@amygracewells
#pseweb
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Why research?
@amygracewells
#pseweb
Slide 5
Slide 5 text
You≠User
@amygracewells
#pseweb
Slide 6
Slide 6 text
Prioritize resources
@amygracewells
#pseweb
Slide 7
Slide 7 text
Reduce problems
@amygracewells
#pseweb
Slide 8
Slide 8 text
What is this diary study
thing you speak of?
@amygracewells
#pseweb
Slide 9
Slide 9 text
–Nielsen Norman Group
“A diary study is a research method used to
collect qualitative data about user behaviors,
activities, and experiences over time.”
Slide 10
Slide 10 text
Key attributes
• Qualitative
• Self-reported at regular
intervals
• Contextual understanding
of user behaviors
• Good for “how” questions
@amygracewells
#pseweb
Slide 11
Slide 11 text
Understand behavior
• Habits — time of day; how they choose to share
• Usage scenarios — primary tasks; workflows for completing longer-
term tasks (test these later in the process)
• Attitudes and motivations — motivation to perform specific tasks;
users feeling and thinking
• Changes in behaviors and perceptions — learnable; loyalty; perception
• Customer journeys — cross-channel user experience across different
devices and channels; cumulative effect of multiple touchpoints
@amygracewells
#pseweb
Slide 12
Slide 12 text
Why diary studies?
@amygracewells
#pseweb
Slide 13
Slide 13 text
The “poor man’s field study”
aka perfectly designed for higher ed
@amygracewells
#pseweb
Slide 14
Slide 14 text
Learnability & Performance
@amygracewells
#pseweb
Slide 15
Slide 15 text
Pros
• Immediate reaction, not remembering
• Test systems that are used regularly
• Effect of environment
• More in-depth than a single usability study
@amygracewells
#pseweb
Slide 16
Slide 16 text
Cons
• No observation
• Complicated and time-intensive to set up
• Retaining participants can be difficult
• Qualitative results take more time to interpret
@amygracewells
#pseweb
Slide 17
Slide 17 text
When to use diary studies?
@amygracewells
#pseweb
Slide 18
Slide 18 text
Diary studies to
• Track long processes over several days (finding and
registering for classes, reserving campus space)
• Discover what motivates users to initiate certain
actions (using search bar)
• Determine how a system factors into regular habits
• Assess retention and/or learnability
@amygracewells
#pseweb
Slide 19
Slide 19 text
How to do a diary study
@amygracewells
#pseweb
Slide 20
Slide 20 text
Include
• Clear instructions
• Template
• Collection method
• Rewards/Encouragement
@amygracewells
#pseweb
Slide 21
Slide 21 text
Disclaimer
Know your university rules (or don’t).
Some may require approval of research offices/IRB.
@amygracewells
Slide 22
Slide 22 text
Instructions
• Be explicit
• When/how often should they report
• How they should report
• What they should report
• Provide help
• Direct contact info
@amygracewells
#pseweb
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
Template
• Gets the info you want with a question-based template
• Makes it easier for the participant
• Allows for limited quantitative questions
@amygracewells
#pseweb
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
Allow space for activities,
thoughts, and frustrations
@amygracewells
#pseweb
Slide 27
Slide 27 text
Collection method
• Email
• Evernote
• SMS
• SurveyMonkey
• Google Drive (doc or form)
• UserTesting
• Voicemails
@amygracewells
#pseweb
Slide 28
Slide 28 text
Collection consideration
• Audience
• Environment
• Comfort with technology
• Allow for media
@amygracewells
#pseweb
Slide 29
Slide 29 text
Retention
• Choose rewards that fit your budget (gift cards, swag)
• Set milestones for rewards and do your part
• Send reminders and thanks
@amygracewells
#pseweb
Slide 30
Slide 30 text
Don’t miss post-study
opportunities
Conduct moderated interviews to discuss overall experience.
Clarify any details or ask any questions from the results.
@amygracewells
#pseweb
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
How do we recruit?
@amygracewells
#pseweb
Slide 33
Slide 33 text
Choose wisely
• Goal is 4-6 participants
• Choose 10-12 from target audience
• Conduct interviews (bonus if you can provide incentive for this)
• Select those that answer freely and most likely to have
time to participate (aka don’t choose an athlete during peak season)
@amygracewells
#pseweb
Slide 34
Slide 34 text
Set up success
• Conduct a training
• Provide feedback after the first submission (but keep it
minimal; you’re not here to bias)
• Thank them after submission
• Remind about upcoming reward milestones
@amygracewells
#pseweb
Slide 35
Slide 35 text
How do we use results?
@amygracewells
#pseweb
Slide 36
Slide 36 text
No content
Slide 37
Slide 37 text
Qualitative results require
time, time, & more time
Look for patterns
Color-code
Save repeated words and phrases
@amygracewells
#pseweb
Slide 38
Slide 38 text
Map their experience
Grab the post-its
Map their key tasks
Map their obstacles
Map their emotions
@amygracewells
#pseweb
Slide 39
Slide 39 text
Let their words
make the impact
Include quotes in results report
@amygracewells
#pseweb
Slide 40
Slide 40 text
Results inform
• Future testing by identifying issues
• Iterative testing such as usability studies
• Resource allocation
• Case for more testing and testing budgets
@amygracewells
#pseweb
Slide 41
Slide 41 text
Thanks for listening
Amy Grace Wells
Editorial Director, UX Booth
[email protected]
@amygracewells
Slide 42
Slide 42 text
Sources
• Nielsen Norman Group
• User Testing
• UXMastery
• UX Booth
• Images from: 123rf, Pinterest, BlinkUX, N/N Group, Spotless