UX Lisbon Remote Research Workshop

Slides from May 17th, 2012 Remote Research workshop at UX Lisbon.

Transcript

1. Remote Research Workshop
2. Photo Credit: recados-orkut.net
3. Remote Design Research Definition: Research where the participant is not sharing physical space with the researcher. In person Remote % of qualitative research done remotely* * Source: My wild guess
4. #UXLX #RR
5. Bayesian methods Animated pedagogical agency 1995 Infer when users needed help Decision Theory & Adaptive Systems Group
7. Misunderstood animated pedagogical agent or spawn of Satan? ” “
8. Bad Research
9. 1) Academic Focus 2) Lab-based 3) False Premise
10. THAT IS WHY WE WENT REMOTE
11. Btw I’m Nate Bolt
12. 12 YEARS 293 STUDIES 3,108 PARTICIPANTS (MODERATED) 118 ORGANIZATIONS 21,610 HOURS
13. “SOCIAL IMPACT OF DIGITAL TECHNOLOGY”
14. A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE
15. SF to PARIS IN TWO MINUTES
17. jumboltron.com
18. OMG PLEASE BUY THIS BOOK
19. Let’s jump right in
20. What’s New with UX Research Tools?
21. BUT FIRST
22. Moderated versus Automated
23. Moderated versus Automated 90% 10%
24. Moderated versus Automated Photo Credit: singularityhub.com
25. Integrated Recording That’s really awesome GoToMeeting 5.2 Moderated
26. Hugging Moderated Photo Credit: Diane Loviglio, Mozilla At Mozilla and MailChimp
27. Usabilla Automated Mobile
28. HOW TO BE graceful When All Your TECHNOLOGY BREAKS
29. Follow the buddy system
30. It’s laptop time
31. remoteresear.ch
32. remoteresear.ch/lisbon
33. (BROWSER TIME)
34. IN CASE THE INTERNET FAILS Research experience Most important thing to get out of today? Biggest fear or skepticism about remote research?
35. Thanks, Volunteer!
36. Four Takeaways 1. See moderated method & recording with GoToMeeting 2.Automated with Usabilla 3.Self-moderated tablet hugging 4.Quick incentive payment
37. Eff you sales guy
38. WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results Pitfalls
40. Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do in action movies?
41. How much practical research experience do I need to have for this workshop? ” - Many People “
42. Will this be a long lecture or an actual workshop? ” - Some Guy on Twitter “
43. Can we do contextual inquiries remotely? ” - Nick Dunlavey | @ndunlavey “
44. The physical and computing ecosystem
45. Now we do a little experiment
46. EXERCISE #1 “Giraffe” Binder PDF Page 4 Table groups Pick one moderator and one participant (they need to be in different locations) 10 minutes of moderating
47. remoteresear.ch/lisbon “The Workshop Binder”
48. MODERATOR Instructions Your participant has a sheet of paper and that’s it Goal: find out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.
49. It’s A RACE You only have 10 minutes to do this max.
50. Surprises?
51. Was there emotion?
52. WHEN DOES REMOTE WIN?
53. Marv. Creative director UX designer product manager developer IA co-founder.
54. Marv’s Boss. He’s always on a plane and loves birds.
55. Hey Marv, I want to save some money on research.
56. NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.
57. TIME-AWARE RESEARCH IS THE BEST
58. I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE
59. INVOLVED OBSERVERS Easier to collaborate in real time.
60. TITLE TEXT GEOGRAPHY
61. WHEN DOES REMOTE LOSE?
63. Hey Marv, I want feedback on our MMO video game from 13yo English speaking kids in France & Latin America with webcams.
64. I guess, Boss. Might be hard to do with remote testing.
65. BANDWIDTH Yes, it’s an issue.
66. TRANSLATION It’s expensive and difficult to do live.
67. MINORS Consent is a bear. Especially if they’re under 13.
68. FACES Most users don’t have webcams. Webcams are a little invasive.
69. YOUR BOSS
70. SUPER DUPER SECURITY
71. AN EPIC FAIL.
73. Bad Research We basically did
74. Kids will lie for coins People mis-assessed their English levels Bandwidth was really a problem Kids don’t play when their parents are home
75. “ What happens when you let go of a balloon? ” HOW TO FIND ACTUAL SPEAKERS OF YOUR LANGUAGE IN ANY COUNTRY: “ Qu'est-ce qui se passe quand vous lâchez un ballon? ”
76. DESIGNING REMOTE STUDIES
77. Yo Marv, get me some insight on WHY people are bailing out of checkout.
78. Sure, Boss. Let’s combine methods. Intercept & observe 10 users live, and do 10 self- moderated with usertesting.com
79. TIME-AWARE TASKS Told to find purple pumps. Doesn’t care. Has to wear this for sister’s wedding this weekend. Whole different interview.
80. HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.
81. REAL TIME OBSERVERS
83. TITLE TEXT
84. PROTOTYPES, WIREFRAMES, SKETCHES?
85. HECK YES.
87. RECRUITING FOR REMOTE
88. RECRUITING OPTIONS Ethnio Links or house ads to online forms Recruiting agencies Customer email lists Panels Craigslist/online ads
90. RECRUITING OPTIONS Grab someone in the hallway Grab someone who isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness
91. SITE ISN’T AN OPTION GEO IP MEMBERS ONLY
92. GOLD STANDARD Live recruiting Get as close as you can
93. USE ETHNIO WHEN Website access Some budget Quotas are relatively simple
94. USE FORMS WHEN You can place a link but not code Low budget Quotas are relatively complex Options: Google Docs Wufoo
95. USE AGENCIES WHEN No website access People whine about first-time visitors Higher budget Online recruiting failed
96. CUSTOMER LISTS Sub-segments Ownership requirements No website access
97. PANEL
98. SORT OF
100. CRAIGSLIST When other methods don’t work No dough Takes a while Embarrassing to admit to your friends
101. LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response (low average You Need: 10,000 uniques/day
102. TITLE TEXT
103. remoteresear.ch/samples “Online Consent”
104. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location
105. Screener constraints Keep it short Different from surveys Who gets paid?
106. MODERATED must-haves Consent Tech set-up The beef De-installation Thanks & incentive
107. AUTOMATED must-haves Expectation setting Instructions The beef Thanks & incentive
108. Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current tasks Different: Compatible tech
109. Logistics: What & How Provide test target Access Credentials
110. Logistics: Who & When How to connect with the user Managing time zones Session length
111. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location
112. STAGED Incentives For Contextual Inquiry Start with $75, then at the end offer $25 more for environment photos Self-made diary studies
114. Advanced planning: hybrid studies Why use only one method? Some methods don’t tell you the why 3-5 remote interviews is all you need
115. EXERCISE #2 Binder PDF Page 6 Use ethn.io or dev.ethn.io (beta) or wufoo Under 8 Questions Advanced Bonus: create an automated and moderated screener okay, draft a screener
116. remoteresear.ch/lisbon “The Workshop Binder”
117. Marv-o - what in the world does automated/ unmoderated/ asynchronous research mean?
118. Sure, boss. There is an AVALANCHE of UX TOOLS.
119. “At least a little behavioral” No surveys. No standard web analytics.
120. Since 2007
121. Can tools make us better researchers?
122. CATEGORIES 5
123. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
124. Tasks On Concepts For A Long Time People Talk To A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site
125. I.A. & Navigation For Ad Agencies Ammunition Insight Just Better Analytics Still More Ammunition More Ammunition
126. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live Deep Behavioral UX Insight
127. Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
128. Ad Agencies Love It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
129. Jared Spool Hates It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
131. Mobile Conceptual For A Long Time Self-Moderated Moderated Automated Static Automated Live }
132. Remote Research Toolkit UNMODERATED (AKA AUTOMATED)
133. 3 parts Provide test target Record user behavior Capture comments (optional)
134. USABILLA Clicks & Questions Heatmaps You do your own recruiting $49 - $199
137. TITLE TEXT LOOP11 (UN-MODERATED)
138. TITLE TEXT
139. GOTOMEETING - $39/MO
142. TITLE TEXT
149. WHAT WE USE. A LOT. • Ethnio + GTM • Usabilla + Usertesting.com • Guerrilla mobile
150. EXERCISE #3 usertesting.com - 1 free user CODE: uxlisbon2012 usabilla - 2 months free (Sign up for an account, but cancel on the payment page. Send an email to support@usabilla.com and mention UXLX) ethnio - 1 month free or trial. CODE: WORKSHOPLOVE Either “unmoderated” or “practice interviewing”
152. RESULTS
153. REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS
154. WHAT THEY WORRY ABOUT
155. WHO ARE YOU AND CAN I TRUST YOU?
156. IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY FACE?
157. WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?
158. WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER AT THE END?
159. WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?
160. IS THIS GOING TO END UP ON YOUTUBE?
161. OXYMORON? REMOTE EMPATHY
162. Taking Notes Second monitor if possible Excel Hack (remoteresear.ch/samples)
163. Observer Involvement IM Campfire (!) Integrated Chat Passing Notes
164. HYBRID STUDIES Numbers actually do lie sometimes And they don’t tell you the why 3-5 remote interviews is all you need
165. So, what could go wrong?
166. 3 points of failure Technology yours participant’s observer’s Recruiting Test Design
167. Control what you can control Which is, your set up. You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.
168. Prepare for what you can’t control User’s dog trips over power cord Internet affected by local weather User can’t install Doesn’t know how to install
169. Mid-Session Tech Problems
170. Live Recruiting Problems What if they gave a study and nobody came?
171. Increase exposure When you’re not getting enough views When the wrong type of users are responding
172. Increase incentives When it’s cheaper When you’re almost getting enough When segments are tough
173. Switch methods When base traffic level isn’t high enough
174. Script Problems Cut the BS.
175. ADVANCED TECHNIQUES One to many moderation Multi-threading
176. PORTABLE RESEARCH
178. BURN THE SCRIPT FIND OUT WHAT MATTERS.
179. ADVANCED TECHNIQUES High Emotions
180. TITLE TEXT
184. THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE
185. DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the view finder, silly •Actual duct tape
187. DISCOUNTS!! Usertesting.com 1 free user. uxlisbon2012 Usabilla 2 months free of any plan. Sign up for an account, but cancel on the payment page. Send an email to support@usabilla.com and mention UXLX Ethnio 1 month free any plan. WORKSHOPLOVE
188. remoteresear.ch/lisbon (look in the slides) DISCOUNTS!! (for, well, that last slide)
189. nate@boltpeters.com @boltron