Remote Research Workshop

Slides from the IA Summit 2012 workshop on Remote UX Research

Transcript

1. Photo Credit: Werner Schnell on Flickr
2. REMOTE RESEARCH Definition: Research where the participant is not sharing physical space with the researcher. In person Remote % of qualitative research done remotely* *Source: BP’s best guess
3. #IAS12 #RR
4. Bayesian methods Animated pedagogical agency 1995 Infer when users needed help Decision Theory & Adaptive Systems Group
6. “Misunderstood animated pedagogical agent or spawn of Satan?”
7. Bad Research
8. 1) Academic Focus 2) Lab-based 3) False Premise
9. THAT IS WHY WE WENT REMOTE
10. Let’s jump right in.
11. HOW TO BE graceful When All Your TECHNOLOGY BREAKS
12. Follow the buddy system
13. It’s laptop time
14. remoteresear.ch
15. IN CASE THE INTERNET FAILS Research experience Most important thing to get out of today? Biggest fear or skepticism about remote research?
16. “How much practical research experience do I need to have for this workshop?” - Many People
17. 10 YEARS 244 STUDIES 2,915 PARTICIPANTS (MODERATED) 95 ORGANIZATIONS 19,610 HOURS
18. “SOCIAL IMPACT OF DIGITAL TECHNOLOGY”
19. OMG PLEASE BUY THIS BOOK
20. 60% PROTOTYPE RESEARCH
21. A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE
24. “Can we do contextual inquiries remotely?” - Nick Dunlavey | @ndunlavey
25. The physical and computing ecosystem
27. Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do in action movies?
28. WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results Pitfalls
29. Now we do a little experiment
30. EXERCISE JUMP IN THE POOL Table groups Pick one moderator and one participant (they need to be in different locations) 10 minutes of moderating
31. MODERATOR Instructions Your participant has a sheet of paper and that’s it Goal: find out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.
32. It’s A RACE First participant to make a beautiful oragami giraffe is the winner
33. Up to 10 Minutes
34. Surprises?
35. Was there emotion?
36. WHEN DOES REMOTE WIN?
37. Marv. Creative director UX designer product manager developer IA co-founder.
38. Marv’s Boss. He’s always on a plane and loves birds.
39. Hey Marv, I want to save some money on research.
40. NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.
41. TIME-AWARE RESEARCH IS THE BEST
43. I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE
44. INVOLVED OBSERVERS Easier to collaborate in real time.
45. TITLE TEXT GEOGRAPHY
46. WHEN DOES REMOTE LOSE?
48. Hey Marv, I want feedback on our MMO video game from 13yo English speaking kids in France & Latin America with webcams.
49. I guess, Boss. Might be hard to do with remote testing.
50. BANDWIDTH Yes, it’s an issue.
51. TRANSLATION It’s expensive and difficult to do live.
52. MINORS Consent is a bear. Especially if they’re under 13.
53. FACES Most users don’t have webcams. Webcams are a little invasive.
54. YOUR BOSS
55. SUPER DUPER SECURITY
56. AN EPIC FAIL.
58. “What happens when you let go of a balloon?” HOW TO FIND ACTUAL ENGLISH SPEAKERS
59. Bad Research We basically did
60. Kids will lie for coins People mis-assessed their English levels Bandwidth was really a problem Kids don’t play when their parents are home
61. DESIGNING REMOTE STUDIES
62. Yo Marv, get me some insight on WHY people are bailing out of checkout.
63. Sure, Boss. Let’s combine methods. Intercept & observe 10 users live, and do 10 self- moderated with usertesting.com
64. TIME-AWARE TASKS Told to find purple pumps. Doesn’t care. Has to wear this for sister’s wedding this weekend. Whole different interview.
65. HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.
66. REAL TIME OBSERVERS
68. TITLE TEXT
69. PROTOTYPES, WIREFRAMES, SKETCHES?
70. HECK YES.
72. okay, plan these things
    Method Recruiting Logistics
73. MODERATED must-haves Consent Tech set-up The beef De-installation Thanks & incentive
74. UNMODERATED must-haves Expectation setting Instructions The beef Thanks & incentive
75. Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current tasks Different: Compatible tech
76. Logistics: What & How Provide test target Access Credentials
77. Logistics: Who & When How to connect with the user (we’ll discuss in Tools) Managing time zones Session length
78. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location
79. STAGED Incentives For Contextual Inquiry Start with $75, then at the end offer $25 more for environment photos Self-made diary studies
81. Advanced planning: hybrid studies Why use only one method? Some methods don’t tell you the why 3-5 remote interviews is all you need
82. RECRUITING FOR REMOTE
83. So Marv, how do we find people for these studies?
84. Stop harassing me and get off that weird plane.
85. RECRUITING OPTIONS Grab someone in the hallway Grab someone who isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness
86. GOLD STANDARD Live recruiting Get as close as you can
87. RECRUITING OPTIONS Ethnio Links or house ads to online forms Recruiting agencies Customer email lists Panels Craigslist/online ads
88. USE ETHNIO WHEN Website access Some budget Quotas are relatively simple
89. USE FORMS WHEN You can place a link but not code Low budget Quotas are relatively complex Options: Google Docs Wufoo
90. USE AGENCIES WHEN No website access People whine about first-time visitors Higher budget Online recruiting failed
91. CUSTOMER LISTS Sub-segments Ownership requirements No website access
92. PANEL
93. SORT OF
95. CRAIGSLIST When other methods don’t work No dough Takes a while Embarrassing to admit to your friends
96. LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response (low average You Need: 10,000 uniques/day
98. FUCK TIME SHARE SALES PITCHES
100. TITLE TEXT
101. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location
102. Screener constraints Keep it short Different from surveys Who gets paid?
103. FIRE? see you at 4pm
105. Your best questions
106. Marv-o - what in the world does automated/ unmoderated/ asynchronous research mean?
107. Sure, boss. There is an AVALANCHE of UX TOOLS.
108. “At least a little behavioral” No surveys. No standard web analytics.
109. Since 2007
110. Can tools make us better researchers?
111. CATEGORIES 5
112. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
113. Tasks On Concepts For A Long Time People Talk To A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site
114. I.A. & Navigation For Ad Agencies Ammunition Insight Just Better Analytics Still More Ammunition More Ammunition
115. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live Deep Behavioral UX Insight
116. Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
117. Ad Agencies Love It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
118. Jared Spool Hates It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live
120. Remote Research Toolkit UNMODERATED (AKA AUTOMATED)
121. 3 parts Provide test target Record user behavior Capture comments (optional)
122. USABILLA Clicks & Questions Heatmaps You do your own recruiting $49 - $199
125. TITLE TEXT LOOP11 (UN-MODERATED)
126. TITLE TEXT
127. GOTOMEETING - $39/MO
130. TITLE TEXT
137. WHAT WE USE. A LOT. • Ethnio + GTM + iShowYouHD • Usabilla + Loop11 + Usertesting.com
138. EXERCISE usertesting.com - code: uxlondon (1 free test)
140. RESULTS
141. REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS
142. WHAT THEY WORRY ABOUT
143. WHO ARE YOU AND CAN I TRUST YOU?
144. IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY FACE?
145. WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?
146. WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER AT THE END?
147. WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?
148. IS THIS GOING TO END UP ON YOUTUBE?
149. OXYMORON? REMOTE EMPATHY
150. Taking Notes Second monitor if possible Excel Hack (for time codes) Reframerapp.com
151. How to rig your notes Leave one column (A) empty for notes. Set the cells of the next column (B) to record the clock time of each entry in A. =IF(A2=“”,“”,IF(B2=“”,NOW(),B2)) Set the cells of the next column (C) to record the difference between the first entry and the current entry =IF(ISERROR(B2-$B$2),“”,B2-$B$2) Fill down (Ctrl-D). Expect to use up to 120 rows for a session. Hide column B. You’ll never need to see it What the formulas look like Auto timestamp! Don’t write this down, it’s on page 104
152. Observer Involvement IM Campfire Integrated Chat Passing Notes
153. Recording Tools Phone Patch Cables Hardware Camtasia (Windows) iShowU HD (Mac) Software
154. Phone patch set up
156. HYBRID STUDIES Numbers actually do lie sometimes And they don’t tell you the why 3-5 remote interviews is all you need
157. So, what could go wrong?
158. 3 points of failure Technology yours participant’s observer’s Recruiting Test Design
159. Control what you can control Which is, your set up. You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.
160. Prepare for what you can’t control User’s dog trips over power cord Internet affected by local weather User can’t install Doesn’t know how to install
161. Mid-Session Tech Problems
162. Live Recruiting Problems What if they gave a study and nobody came?
163. Increase exposure When you’re not getting enough views When the wrong type of users are responding
164. Increase incentives When it’s cheaper When you’re almost getting enough When segments are tough
165. Switch methods When base traffic level isn’t high enough
166. Script Problems Cut the BS.
167. MOBILE REMOTE RESEARCH?
169. ADVANCED TECHNIQUES One to many moderation Multi-threading
170. PORTABLE RESEARCH
172. BURN THE SCRIPT FIND OUT WHAT MATTERS.
173. ADVANCED TECHNIQUES High Emotions
174. TITLE TEXT
175. TITLE TEXT
179. THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE
180. DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the view finder, silly •Actual duct tape
182. Tony Fadell
183. Treat every member of your team like an artist.
184. Logic ≠ Intuition
185. Logical photo by flickr.com/photos/80849382@N00/3306939463/
186. Intuitive photo by flickr.com/photos/51035750608@N01/117683322/
187. How do you know when something is innovative?
188. “It’s been done before” - Bill Buxton | “Long nose of innovation”
191. “It’s scary” - Marty Neumeier | The Designful Company
192. remoteresear.ch/ias12 DISCOUNTS!!
193. nate@boltpeters.com @boltron