Webvisions Barcelona Remote Research Workshop

Slides from my workshop on July 5th, 2012 at Webvisions Barcelona. http://webvisions.com/barcelona

1. Remote Research Workshop
2. Photo Credit: kaboodle.com
3. Remote Design Research Definition: Research where the participant is not sharing physical space with the researcher. In person Remote % of qualitative research done remotely* * Source: My wild guess
4. #WEBVISIONS #RR
5. Decision Theory & Adaptive Systems Group
7. Misunderstood animated pedagogical agent or spawn of Satan? ” “
8. Bad Research
9. 1) Academic Focus 2) Lab-based 3) False Premise
10. THAT IS WHY WE WENT REMOTE
11. Btw I’m Nate Bolt
12. 12 YEARS 293 STUDIES 3,108 PARTICIPANTS (MODERATED) 118 ORGANIZATIONS 21,610 HOURS
15. “SOCIAL IMPACT OF DIGITAL TECHNOLOGY”
16. A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE
17. SF to PARIS IN TWO MINUTES
19. london to iceland in two minutes
21. Let’s jump right in
22. What’s New with UX Research Tools?
23. A DISTINCTION
24. Moderated versus Automated
25. Moderated versus Automated 90% 10%
26. Moderated versus Automated Photo Credit: singularityhub.com
27. Integrated Recording That’s really awesome GoToMeeting 5.2 Moderated
28. Hugging Moderated Photo Credit: Diane Loviglio, Mozilla At Mozilla and MailChimp
29. Usabilla Automated Mobile
30. HOW TO BE graceful When All Your TECHNOLOGY BREAKS
31. Follow the buddy system
32. It’s laptop time
33. remoteresear.ch or dev.ethn.io/40762
34. remoteresear.ch/barcelona
35. (BROWSER TIME)
36. IN CASE THE INTERNET FAILS Research experience Most important thing to get out of today? Biggest fear or skepticism about remote research?
37. Thanks, Volunteer!
38. Four Takeaways 1. See moderated method & recording with GoToMeeting 2.Automated with Usabilla 3.Self-moderated tablet hugging 4.Quick incentive payment
39. Eff you sales guy
40. WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results Pitfalls
43. Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do in action movies?
44. How much practical research experience do I need to have for this workshop? ” - Many People “
45. Will this be a long lecture or an actual workshop? ” - Some Guy on Twitter “
46. Can we do contextual inquiries remotely? ” - Nick Dunlavey | @ndunlavey “
47. The physical and computing ecosystem
48. Now we do a little experiment
49. EXERCISE #1 “Giraffe” Binder PDF Page 4 Table groups Pick one moderator and one participant (they need to be in different locations) 10 minutes of moderating
50. MODERATOR Instructions Your participant has a sheet of paper and that’s it Goal: find out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.
51. It’s A RACE You only have 10 minutes to do this max.
52. Surprises?
53. Was there emotion?
54. WHEN DOES REMOTE WIN?
55. Marv. Creative director UX designer product manager developer IA co-founder.
56. Marv. Creative director UX designer product manager developer IA co-founder.
57. Marv’s Boss. He’s always on a plane and loves birds.
58. Marv’s Boss. He’s always on a plane and loves birds.
59. Hey Marv, I want to save some money on research.
60. NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.
61. TIME-AWARE RESEARCH IS THE BEST
62. I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE
63. INVOLVED OBSERVERS Easier to collaborate in real time.
64. TITLE TEXT GEOGRAPHY
65. WHEN DOES REMOTE LOSE?
67. Hey Marv, I want feedback on our MMO video game from 13yo English speaking kids in France & Latin America with webcams.
68. I guess, Boss. Might be hard to do with remote testing.
69. BANDWIDTH Yes, it’s an issue.
70. TRANSLATION It’s expensive and difficult to do live.
71. MINORS Consent is a bear. Especially if they’re under 13.
72. FACES Most users don’t have webcams. Webcams are a little invasive.
73. YOUR BOSS
74. SUPER DUPER SECURITY
75. AN EPIC FAIL.
78. Bad Research We basically did
79. Kids will lie for coins People mis-assessed their English levels Bandwidth was really a problem Kids don’t play when their parents are home
80. “ What happens when you let go of a balloon? ” HOW TO FIND ACTUAL SPEAKERS OF YOUR LANGUAGE IN ANY COUNTRY: “ Qu'est-ce qui se passe quand vous lâchez un ballon? ”
81. DESIGNING REMOTE STUDIES
82. Yo Marv, get me some insight on WHY people are bailing out of checkout.
83. Sure, Boss. Let’s combine methods. Intercept & observe 10 users live, and do 10 self- moderated with usertesting.com
84. TIME-AWARE TASKS Told to find purple pumps. Doesn’t care. Has to wear this for sister’s wedding this weekend. Whole different interview.
85. HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.
86. REAL TIME OBSERVERS
89. TITLE TEXT
90. PROTOTYPES, WIREFRAMES, SKETCHES?
91. HECK YES.
93. RECRUITING FOR REMOTE
94. RECRUITING OPTIONS Ethnio Links or house ads to online forms Recruiting agencies Customer email lists Panels Craigslist/online ads
98. RECRUITING OPTIONS Grab someone in the hallway Grab someone who isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness
99. SITE ISN’T AN OPTION GEO IP MEMBERS ONLY
100. GOLD STANDARD Live recruiting Get as close as you can
101. USE ETHNIO WHEN Website access Some budget Quotas are relatively simple
102. USE FORMS WHEN You can place a link but not code Low budget Quotas are relatively complex Options: Google Docs Wufoo
103. USE AGENCIES WHEN No website access People whine about first-time visitors Higher budget Online recruiting failed
104. CUSTOMER LISTS Sub-segments Ownership requirements No website access
105. PANEL
106. SORT OF
108. CRAIGSLIST When other methods don’t work No dough Takes a while Embarrassing to admit to your friends
109. LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response (low average You Need: 10,000 uniques/day
110. TITLE TEXT
111. remoteresear.ch/samples “Online Consent”
112. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location
113. Screener constraints Keep it short Different from surveys Who gets paid?
114. MODERATED must-haves
115. MODERATED must-haves Consent
116. MODERATED must-haves Consent Tech set-up
117. MODERATED must-haves Consent Tech set-up The beef
118. MODERATED must-haves Consent Tech set-up The beef De-installation
119. MODERATED must-haves Consent Tech set-up The beef De-installation Thanks & incentive
120. AUTOMATED must-haves
121. AUTOMATED must-haves Expectation setting
122. AUTOMATED must-haves Expectation setting Instructions
123. AUTOMATED must-haves Expectation setting Instructions The beef
124. AUTOMATED must-haves Expectation setting Instructions The beef Thanks & incentive
125. Recruiting
126. Recruiting Same: Good “talkers” Same: Demographic fits
127. Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current tasks Different: Compatible tech
128. Logistics: What & How Provide test target Access Credentials
129. Logistics: Who & When How to connect with the user Managing time zones Session length
130. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location
131. STAGED Incentives For Contextual Inquiry Start with $75, then at the end offer $25 more for environment photos Self-made diary studies
133. Advanced planning: hybrid studies Why use only one method? Some methods don’t tell you the why 3-5 remote interviews is all you need
134. EXERCISE #2 okay, draft a screener
135. EXERCISE #2 Binder PDF Page 6 Use ethn.io or dev.ethn.io (beta) or wufoo Under 8 Questions okay, draft a screener
136. EXERCISE #2 Binder PDF Page 6 Use ethn.io or dev.ethn.io (beta) or wufoo Under 8 Questions Advanced Bonus: create an automated and moderated screener okay, draft a screener
138. Marv-o - what tools should we pick? I mean, tell me to use for pete’s ake
139. Sure, boss. There is an AVALANCHE of UX TOOLS.
140. “At least a little behavioral” No surveys. No standard web analytics.
142. Since 2007
143. Can tools make us better researchers?
144. CATEGORIES 5
145. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
146. Tasks On Concepts For A Long Time People Talk To A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site
147. I.A. & Navigation For Ad Agencies Ammunition Insight Just Better Analytics Still More Ammunition More Ammunition
148. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live Deep Behavioral UX Insight
149. Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
150. Ad Agencies Love It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
151. Jared Spool Hates It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
152. Jared Spool Hates It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
154. Mobile Conceptual For A Long Time Self-Moderated Moderated Automated Static Automated Live }
155. Remote Research Toolkit UNMODERATED (AKA AUTOMATED)
156. 3 parts Provide test target Record user behavior Capture comments (optional)
157. USABILLA Clicks & Questions Heatmaps You do your own recruiting $49 - $199
162. TITLE TEXT LOOP11 (UN-MODERATED)
163. TITLE TEXT
164. GOTOMEETING - $39/MO
167. TITLE TEXT
174. WHAT WE USE. A LOT. • Ethnio + GTM • Usabilla + Usertesting.com • Silverback (Mac) & Camtasia (PC) • Guerrilla mobile
176. RESULTS
177. REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS
178. WHAT THEY WORRY ABOUT
179. WHO ARE YOU AND CAN I TRUST YOU?
180. IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY FACE?
181. WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?
182. WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER AT THE END?
183. WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?
184. IS THIS GOING TO END UP ON YOUTUBE?
185. OXYMORON? REMOTE EMPATHY
186. Taking Notes Second monitor if possible Excel Hack (remoteresear.ch/samples)
187. Observer Involvement IM Campfire (!) Integrated Chat Passing Notes
188. HYBRID STUDIES Numbers actually do lie sometimes And they don’t tell you the why 3-5 remote interviews is all you need
189. So, what could go wrong?
190. 3 points of failure Technology yours participant’s observer’s Recruiting Test Design
191. Control what you can control Which is, your set up. You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.
192. Prepare for what you can’t control User’s dog trips over power cord Internet affected by local weather User can’t install Doesn’t know how to install
193. Mid-Session Tech Problems
194. Live Recruiting Problems What if they gave a study and nobody came?
195. Increase exposure When you’re not getting enough views When the wrong type of users are responding
196. Increase incentives When it’s cheaper When you’re almost getting enough When segments are tough
197. Switch methods When base traffic level isn’t high enough
198. Script Problems Cut the BS.
199. ADVANCED TECHNIQUES One to many moderation Multi-threading
200. PORTABLE RESEARCH
202. BURN THE SCRIPT FIND OUT WHAT MATTERS.
203. ADVANCED TECHNIQUES High Emotions
204. TITLE TEXT
208. THE FUTURE
209. THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE
210. DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the view finder, silly •Actual duct tape
212. nate@boltron.com @boltron