Remote Research Workshop

Transcript

1. Photo Credit: Werner Schnell on Flickr

2. REMOTE RESEARCH Definition: Research where the participant is not sharing

   physical space with the researcher. In person Remote % of qualitative research done remotely* *Source: BP’s best guess

3. #IAS12 #RR

4. Bayesian methods Animated pedagogical agency 1995 Infer when users needed

   help Decision Theory & Adaptive Systems Group
5. None

6. “Misunderstood animated pedagogical agent or spawn of Satan?”

7. Bad Research

8. 1) Academic Focus 2) Lab-based 3) False Premise

9. THAT IS WHY WE WENT REMOTE

10. Let’s jump right in.

11. HOW TO BE graceful When All Your TECHNOLOGY BREAKS

12. Follow the buddy system

13. It’s laptop time

14. remoteresear.ch

15. IN CASE THE INTERNET FAILS Research experience Most important thing

    to get out of today? Biggest fear or skepticism about remote research?

16. “How much practical research experience do I need to have

    for this workshop?” - Many People

17. 10 YEARS 244 STUDIES 2,915 PARTICIPANTS (MODERATED) 95 ORGANIZATIONS 19,610

    HOURS

18. “SOCIAL IMPACT OF DIGITAL TECHNOLOGY”

19. OMG PLEASE BUY THIS BOOK

20. 60% PROTOTYPE RESEARCH

21. A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE

22. None
23. None

24. “Can we do contextual inquiries remotely?” - Nick Dunlavey |

    @ndunlavey

25. The physical and computing ecosystem

26. None

27. Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do

    in action movies?

28. WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results

    Pitfalls

29. Now we do a little experiment

30. EXERCISE JUMP IN THE POOL Table groups Pick one moderator

    and one participant (they need to be in different locations) 10 minutes of moderating

31. MODERATOR Instructions Your participant has a sheet of paper and

    that’s it Goal: find out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.

32. It’s A RACE First participant to make a beautiful oragami

    giraffe is the winner

33. Up to 10 Minutes

34. Surprises?

35. Was there emotion?

36. WHEN DOES REMOTE WIN?

37. Marv. Creative director UX designer product manager developer IA co-founder.

38. Marv’s Boss. He’s always on a plane and loves birds.

39. Hey Marv, I want to save some money on research.

40. NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.

41. TIME-AWARE RESEARCH IS THE BEST

42. None

43. I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE

44. INVOLVED OBSERVERS Easier to collaborate in real time.

45. TITLE TEXT GEOGRAPHY

46. WHEN DOES REMOTE LOSE?

47. None

48. Hey Marv, I want feedback on our MMO video game

    from 13yo English speaking kids in France & Latin America with webcams.

49. I guess, Boss. Might be hard to do with remote

    testing.

50. BANDWIDTH Yes, it’s an issue.

51. TRANSLATION It’s expensive and difficult to do live.

52. MINORS Consent is a bear. Especially if they’re under 13.

53. FACES Most users don’t have webcams. Webcams are a little

    invasive.

54. YOUR BOSS

55. SUPER DUPER SECURITY

56. AN EPIC FAIL.

57. None

58. “What happens when you let go of a balloon?” HOW

    TO FIND ACTUAL ENGLISH SPEAKERS

59. Bad Research We basically did

60. Kids will lie for coins People mis-assessed their English levels

    Bandwidth was really a problem Kids don’t play when their parents are home

61. DESIGNING REMOTE STUDIES

62. Yo Marv, get me some insight on WHY people are

    bailing out of checkout.

63. Sure, Boss. Let’s combine methods. Intercept & observe 10 users

    live, and do 10 self- moderated with usertesting.com

64. TIME-AWARE TASKS Told to find purple pumps. Doesn’t care. Has

    to wear this for sister’s wedding this weekend. Whole different interview.

65. HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.

66. REAL TIME OBSERVERS

67. None

68. TITLE TEXT

69. PROTOTYPES, WIREFRAMES, SKETCHES?

70. HECK YES.

71. None

72. okay, plan these things
    Method Recruiting Logistics

73. MODERATED must-haves Consent Tech set-up The beef De-installation Thanks &

    incentive

74. UNMODERATED must-haves Expectation setting Instructions The beef Thanks & incentive

75. Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current

    tasks Different: Compatible tech

76. Logistics: What & How Provide test target Access Credentials

77. Logistics: Who & When How to connect with the user

    (we’ll discuss in Tools) Managing time zones Session length

78. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for

    location

79. STAGED Incentives For Contextual Inquiry Start with $75, then at

    the end offer $25 more for environment photos Self-made diary studies
80. None

81. Advanced planning: hybrid studies Why use only one method? Some

    methods don’t tell you the why 3-5 remote interviews is all you need

82. RECRUITING FOR REMOTE

83. So Marv, how do we find people for these studies?

84. Stop harassing me and get off that weird plane.

85. RECRUITING OPTIONS Grab someone in the hallway Grab someone who

    isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness

86. GOLD STANDARD Live recruiting Get as close as you can

87. RECRUITING OPTIONS Ethnio Links or house ads to online forms

    Recruiting agencies Customer email lists Panels Craigslist/online ads

88. USE ETHNIO WHEN Website access Some budget Quotas are relatively

    simple

89. USE FORMS WHEN You can place a link but not

    code Low budget Quotas are relatively complex Options: Google Docs Wufoo

90. USE AGENCIES WHEN No website access People whine about first-time

    visitors Higher budget Online recruiting failed

91. CUSTOMER LISTS Sub-segments Ownership requirements No website access

92. PANEL

93. SORT OF

94. None

95. CRAIGSLIST When other methods don’t work No dough Takes a

    while Embarrassing to admit to your friends

96. LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response

    (low average You Need: 10,000 uniques/day
97. None

98. FUCK TIME SHARE SALES PITCHES

99. None

100. TITLE TEXT

101. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for

     location

102. Screener constraints Keep it short Different from surveys Who gets

     paid?

103. FIRE? see you at 4pm

104. None

105. Your best questions

106. Marv-o - what in the world does automated/ unmoderated/ asynchronous

     research mean?

107. Sure, boss. There is an AVALANCHE of UX TOOLS.

108. “At least a little behavioral” No surveys. No standard web

     analytics.

109. Since 2007

110. Can tools make us better researchers?

111. CATEGORIES 5

112. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated

     Static Automated Live

113. Tasks On Concepts For A Long Time People Talk To

     A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site

114. I.A. & Navigation For Ad Agencies Ammunition Insight Just Better

     Analytics Still More Ammunition More Ammunition

115. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated

     Static Automated Live Deep Behavioral UX Insight

116. Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics

     Automated Static Automated Live

117. Ad Agencies Love It Conceptual For A Long Time Self-Moderated

     Moderated Web Analytics Automated Static Automated Live

118. Jared Spool Hates It Conceptual For A Long Time Self-Moderated

     Moderated Web Analytics Automated Static Automated Live
119. None

120. Remote Research Toolkit UNMODERATED (AKA AUTOMATED)

121. 3 parts Provide test target Record user behavior Capture comments

     (optional)

122. USABILLA Clicks & Questions Heatmaps You do your own recruiting

     $49 - $199
123. None
124. None

125. TITLE TEXT LOOP11 (UN-MODERATED)

126. TITLE TEXT

127. GOTOMEETING - $39/MO

128. None
129. None

130. TITLE TEXT

131. None
132. None
133. None
134. None
135. None
136. None

137. WHAT WE USE. A LOT. • Ethnio + GTM +

     iShowYouHD • Usabilla + Loop11 + Usertesting.com

138. EXERCISE usertesting.com - code: uxlondon (1 free test)

139. None

140. RESULTS

141. REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS

142. WHAT THEY WORRY ABOUT

143. WHO ARE YOU AND CAN I TRUST YOU?

144. IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY

     FACE?

145. WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?

146. WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER

     AT THE END?

147. WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?

148. IS THIS GOING TO END UP ON YOUTUBE?

149. OXYMORON? REMOTE EMPATHY

150. Taking Notes Second monitor if possible Excel Hack (for time

     codes) Reframerapp.com

151. How to rig your notes Leave one column (A) empty

     for notes. Set the cells of the next column (B) to record the clock time of each entry in A. =IF(A2=“”,“”,IF(B2=“”,NOW(),B2)) Set the cells of the next column (C) to record the difference between the first entry and the current entry =IF(ISERROR(B2-$B$2),“”,B2-$B$2) Fill down (Ctrl-D). Expect to use up to 120 rows for a session. Hide column B. You’ll never need to see it What the formulas look like Auto timestamp! Don’t write this down, it’s on page 104

152. Observer Involvement IM Campfire Integrated Chat Passing Notes

153. Recording Tools Phone Patch Cables Hardware Camtasia (Windows) iShowU HD

     (Mac) Software

154. Phone patch set up

155. None

156. HYBRID STUDIES Numbers actually do lie sometimes And they don’t

     tell you the why 3-5 remote interviews is all you need

157. So, what could go wrong?

158. 3 points of failure Technology yours participant’s observer’s Recruiting Test

     Design

159. Control what you can control Which is, your set up.

     You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.

160. Prepare for what you can’t control User’s dog trips over

     power cord Internet affected by local weather User can’t install Doesn’t know how to install

161. Mid-Session Tech Problems

162. Live Recruiting Problems What if they gave a study and

     nobody came?

163. Increase exposure When you’re not getting enough views When the

     wrong type of users are responding

164. Increase incentives When it’s cheaper When you’re almost getting enough

     When segments are tough

165. Switch methods When base traffic level isn’t high enough

166. Script Problems Cut the BS.

167. MOBILE REMOTE RESEARCH?

168. None

169. ADVANCED TECHNIQUES One to many moderation Multi-threading

170. PORTABLE RESEARCH

171. None

172. BURN THE SCRIPT FIND OUT WHAT MATTERS.

173. ADVANCED TECHNIQUES High Emotions

174. TITLE TEXT

175. TITLE TEXT

176. None
177. None
178. None

179. THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE

180. DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the

     view finder, silly •Actual duct tape
181. None

182. Tony Fadell

183. Treat every member of your team like an artist.

184. Logic ≠ Intuition

185. Logical photo by flickr.com/photos/80849382@N00/3306939463/

186. Intuitive photo by flickr.com/photos/51035750608@N01/117683322/

187. How do you know when something is innovative?

188. “It’s been done before” - Bill Buxton | “Long nose

     of innovation”
189. None
190. None

191. “It’s scary” - Marty Neumeier | The Designful Company

192. remoteresear.ch/ias12 DISCOUNTS!!

193. [email protected] @boltron