Webvisions Barcelona Remote Research Workshop

Transcript

1. Remote Research Workshop

2. Photo Credit: kaboodle.com

3. Remote Design Research Definition: Research where the participant is not

   sharing physical space with the researcher. In person Remote % of qualitative research done remotely* * Source: My wild guess

4. #WEBVISIONS #RR

5. Decision Theory & Adaptive Systems Group

6. None

7. Misunderstood animated pedagogical agent or spawn of Satan? ” “

8. Bad Research

9. 1) Academic Focus 2) Lab-based 3) False Premise

10. THAT IS WHY WE WENT REMOTE

11. Btw I’m Nate Bolt

12. 12 YEARS 293 STUDIES 3,108 PARTICIPANTS (MODERATED) 118 ORGANIZATIONS 21,610

    HOURS
13. None
14. None

15. “SOCIAL IMPACT OF DIGITAL TECHNOLOGY”

16. A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE

17. SF to PARIS IN TWO MINUTES

18. None

19. london to iceland in two minutes

20. None

21. Let’s jump right in

22. What’s New with UX Research Tools?

23. A DISTINCTION

24. Moderated versus Automated

25. Moderated versus Automated 90% 10%

26. Moderated versus Automated Photo Credit: singularityhub.com

27. Integrated Recording That’s really awesome GoToMeeting 5.2 Moderated

28. Hugging Moderated Photo Credit: Diane Loviglio, Mozilla At Mozilla and

    MailChimp

29. Usabilla Automated Mobile

30. HOW TO BE graceful When All Your TECHNOLOGY BREAKS

31. Follow the buddy system

32. It’s laptop time

33. remoteresear.ch or dev.ethn.io/40762

34. remoteresear.ch/barcelona

35. (BROWSER TIME)

36. IN CASE THE INTERNET FAILS Research experience Most important thing

    to get out of today? Biggest fear or skepticism about remote research?

37. Thanks, Volunteer!

38. Four Takeaways 1. See moderated method & recording with GoToMeeting

    2.Automated with Usabilla 3.Self-moderated tablet hugging 4.Quick incentive payment

39. Eff you sales guy

40. WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results

    Pitfalls
41. None
42. None

43. Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do

    in action movies?

44. How much practical research experience do I need to have

    for this workshop? ” - Many People “

45. Will this be a long lecture or an actual workshop?

    ” - Some Guy on Twitter “

46. Can we do contextual inquiries remotely? ” - Nick Dunlavey

    | @ndunlavey “

47. The physical and computing ecosystem

48. Now we do a little experiment

49. EXERCISE #1 “Giraffe” Binder PDF Page 4 Table groups Pick

    one moderator and one participant (they need to be in different locations) 10 minutes of moderating

50. MODERATOR Instructions Your participant has a sheet of paper and

    that’s it Goal: find out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.

51. It’s A RACE You only have 10 minutes to do

    this max.

52. Surprises?

53. Was there emotion?

54. WHEN DOES REMOTE WIN?

55. Marv. Creative director UX designer product manager developer IA co-founder.

56. Marv. Creative director UX designer product manager developer IA co-founder.

57. Marv’s Boss. He’s always on a plane and loves birds.

58. Marv’s Boss. He’s always on a plane and loves birds.

59. Hey Marv, I want to save some money on research.

60. NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.

61. TIME-AWARE RESEARCH IS THE BEST

62. I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE

63. INVOLVED OBSERVERS Easier to collaborate in real time.

64. TITLE TEXT GEOGRAPHY

65. WHEN DOES REMOTE LOSE?

66. None

67. Hey Marv, I want feedback on our MMO video game

    from 13yo English speaking kids in France & Latin America with webcams.

68. I guess, Boss. Might be hard to do with remote

    testing.

69. BANDWIDTH Yes, it’s an issue.

70. TRANSLATION It’s expensive and difficult to do live.

71. MINORS Consent is a bear. Especially if they’re under 13.

72. FACES Most users don’t have webcams. Webcams are a little

    invasive.

73. YOUR BOSS

74. SUPER DUPER SECURITY

75. AN EPIC FAIL.

76. None
77. None

78. Bad Research We basically did

79. Kids will lie for coins People mis-assessed their English levels

    Bandwidth was really a problem Kids don’t play when their parents are home

80. “ What happens when you let go of a balloon?

    ” HOW TO FIND ACTUAL SPEAKERS OF YOUR LANGUAGE IN ANY COUNTRY: “ Qu'est-ce qui se passe quand vous lâchez un ballon? ”

81. DESIGNING REMOTE STUDIES

82. Yo Marv, get me some insight on WHY people are

    bailing out of checkout.

83. Sure, Boss. Let’s combine methods. Intercept & observe 10 users

    live, and do 10 self- moderated with usertesting.com

84. TIME-AWARE TASKS Told to find purple pumps. Doesn’t care. Has

    to wear this for sister’s wedding this weekend. Whole different interview.

85. HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.

86. REAL TIME OBSERVERS

87. None
88. None

89. TITLE TEXT

90. PROTOTYPES, WIREFRAMES, SKETCHES?

91. HECK YES.

92. None

93. RECRUITING FOR REMOTE

94. RECRUITING OPTIONS Ethnio Links or house ads to online forms

    Recruiting agencies Customer email lists Panels Craigslist/online ads
95. None
96. None
97. None

98. RECRUITING OPTIONS Grab someone in the hallway Grab someone who

    isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness

99. SITE ISN’T AN OPTION GEO IP MEMBERS ONLY

100. GOLD STANDARD Live recruiting Get as close as you can

101. USE ETHNIO WHEN Website access Some budget Quotas are relatively

     simple

102. USE FORMS WHEN You can place a link but not

     code Low budget Quotas are relatively complex Options: Google Docs Wufoo

103. USE AGENCIES WHEN No website access People whine about first-time

     visitors Higher budget Online recruiting failed

104. CUSTOMER LISTS Sub-segments Ownership requirements No website access

105. PANEL

106. SORT OF

107. None

108. CRAIGSLIST When other methods don’t work No dough Takes a

     while Embarrassing to admit to your friends

109. LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response

     (low average You Need: 10,000 uniques/day

110. TITLE TEXT

111. remoteresear.ch/samples “Online Consent”

112. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for

     location

113. Screener constraints Keep it short Different from surveys Who gets

     paid?

114. MODERATED must-haves

115. MODERATED must-haves Consent

116. MODERATED must-haves Consent Tech set-up

117. MODERATED must-haves Consent Tech set-up The beef

118. MODERATED must-haves Consent Tech set-up The beef De-installation

119. MODERATED must-haves Consent Tech set-up The beef De-installation Thanks &

     incentive

120. AUTOMATED must-haves

121. AUTOMATED must-haves Expectation setting

122. AUTOMATED must-haves Expectation setting Instructions

123. AUTOMATED must-haves Expectation setting Instructions The beef

124. AUTOMATED must-haves Expectation setting Instructions The beef Thanks & incentive

125. Recruiting

126. Recruiting Same: Good “talkers” Same: Demographic fits

127. Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current

     tasks Different: Compatible tech

128. Logistics: What & How Provide test target Access Credentials

129. Logistics: Who & When How to connect with the user

     Managing time zones Session length

130. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for

     location

131. STAGED Incentives For Contextual Inquiry Start with $75, then at

     the end offer $25 more for environment photos Self-made diary studies
132. None

133. Advanced planning: hybrid studies Why use only one method? Some

     methods don’t tell you the why 3-5 remote interviews is all you need

134. EXERCISE #2 okay, draft a screener

135. EXERCISE #2 Binder PDF Page 6 Use ethn.io or dev.ethn.io

     (beta) or wufoo Under 8 Questions okay, draft a screener

136. EXERCISE #2 Binder PDF Page 6 Use ethn.io or dev.ethn.io

     (beta) or wufoo Under 8 Questions Advanced Bonus: create an automated and moderated screener okay, draft a screener
137. None

138. Marv-o - what tools should we pick? I mean, tell

     me to use for pete’s ake

139. Sure, boss. There is an AVALANCHE of UX TOOLS.

140. “At least a little behavioral” No surveys. No standard web

     analytics.
141. None

142. Since 2007

143. Can tools make us better researchers?

144. CATEGORIES 5

145. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated

     Static Automated Live

146. Tasks On Concepts For A Long Time People Talk To

     A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site

147. I.A. & Navigation For Ad Agencies Ammunition Insight Just Better

     Analytics Still More Ammunition More Ammunition

148. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated

     Static Automated Live Deep Behavioral UX Insight

149. Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics

     Automated Static Automated Live

150. Ad Agencies Love It Conceptual For A Long Time Self-Moderated

     Moderated Web Analytics Automated Static Automated Live

151. Jared Spool Hates It Conceptual For A Long Time Self-Moderated

     Moderated Web Analytics Automated Static Automated Live

152. Jared Spool Hates It Conceptual For A Long Time Self-Moderated

     Moderated Web Analytics Automated Static Automated Live
153. None

154. Mobile Conceptual For A Long Time Self-Moderated Moderated Automated Static

     Automated Live }

155. Remote Research Toolkit UNMODERATED (AKA AUTOMATED)

156. 3 parts Provide test target Record user behavior Capture comments

     (optional)

157. USABILLA Clicks & Questions Heatmaps You do your own recruiting

     $49 - $199
158. None
159. None
160. None
161. None

162. TITLE TEXT LOOP11 (UN-MODERATED)

163. TITLE TEXT

164. GOTOMEETING - $39/MO

165. None
166. None

167. TITLE TEXT

168. None
169. None
170. None
171. None
172. None
173. None

174. WHAT WE USE. A LOT. • Ethnio + GTM •

     Usabilla + Usertesting.com • Silverback (Mac) & Camtasia (PC) • Guerrilla mobile
175. None

176. RESULTS

177. REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS

178. WHAT THEY WORRY ABOUT

179. WHO ARE YOU AND CAN I TRUST YOU?

180. IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY

     FACE?

181. WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?

182. WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER

     AT THE END?

183. WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?

184. IS THIS GOING TO END UP ON YOUTUBE?

185. OXYMORON? REMOTE EMPATHY

186. Taking Notes Second monitor if possible Excel Hack (remoteresear.ch/samples)

187. Observer Involvement IM Campfire (!) Integrated Chat Passing Notes

188. HYBRID STUDIES Numbers actually do lie sometimes And they don’t

     tell you the why 3-5 remote interviews is all you need

189. So, what could go wrong?

190. 3 points of failure Technology yours participant’s observer’s Recruiting Test

     Design

191. Control what you can control Which is, your set up.

     You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.

192. Prepare for what you can’t control User’s dog trips over

     power cord Internet affected by local weather User can’t install Doesn’t know how to install

193. Mid-Session Tech Problems

194. Live Recruiting Problems What if they gave a study and

     nobody came?

195. Increase exposure When you’re not getting enough views When the

     wrong type of users are responding

196. Increase incentives When it’s cheaper When you’re almost getting enough

     When segments are tough

197. Switch methods When base traffic level isn’t high enough

198. Script Problems Cut the BS.

199. ADVANCED TECHNIQUES One to many moderation Multi-threading

200. PORTABLE RESEARCH

201. None

202. BURN THE SCRIPT FIND OUT WHAT MATTERS.

203. ADVANCED TECHNIQUES High Emotions

204. TITLE TEXT

205. None
206. None
207. None

208. THE FUTURE

209. THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE

210. DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the

     view finder, silly •Actual duct tape
211. None

212. [email protected] @boltron