Remote Research Workshop

Remote Research Workshop

Slides from the IA Summit 2012 workshop on Remote UX Research

9bb074de66a96eeacd0b275ee10e57d3?s=128

boltpeters

March 22, 2012
Tweet

Transcript

  1. Photo Credit: Werner Schnell on Flickr

  2. REMOTE RESEARCH Definition: Research where the participant is not sharing

    physical space with the researcher. In person Remote % of qualitative research done remotely* *Source: BP’s best guess
  3. #IAS12 #RR

  4. Bayesian methods Animated pedagogical agency 1995 Infer when users needed

    help Decision Theory & Adaptive Systems Group
  5. None
  6. “Misunderstood animated pedagogical agent or spawn of Satan?”

  7. Bad Research

  8. 1) Academic Focus 2) Lab-based 3) False Premise

  9. THAT IS WHY WE WENT REMOTE

  10. Let’s jump right in.

  11. HOW TO BE graceful When All Your TECHNOLOGY BREAKS

  12. Follow the buddy system

  13. It’s laptop time

  14. remoteresear.ch

  15. IN CASE THE INTERNET FAILS Research experience Most important thing

    to get out of today? Biggest fear or skepticism about remote research?
  16. “How much practical research experience do I need to have

    for this workshop?” - Many People
  17. 10 YEARS 244 STUDIES 2,915 PARTICIPANTS (MODERATED) 95 ORGANIZATIONS 19,610

    HOURS
  18. “SOCIAL IMPACT OF DIGITAL TECHNOLOGY”

  19. OMG PLEASE BUY THIS BOOK

  20. 60% PROTOTYPE RESEARCH

  21. A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE

  22. None
  23. None
  24. “Can we do contextual inquiries remotely?” - Nick Dunlavey |

    @ndunlavey
  25. The physical and computing ecosystem

  26. None
  27. Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do

    in action movies?
  28. WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results

    Pitfalls
  29. Now we do a little experiment

  30. EXERCISE JUMP IN THE POOL Table groups Pick one moderator

    and one participant (they need to be in different locations) 10 minutes of moderating
  31. MODERATOR Instructions Your participant has a sheet of paper and

    that’s it Goal: find out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.
  32. It’s A RACE First participant to make a beautiful oragami

    giraffe is the winner
  33. Up to 10 Minutes

  34. Surprises?

  35. Was there emotion?

  36. WHEN DOES REMOTE WIN?

  37. Marv. Creative director UX designer product manager developer IA co-founder.

  38. Marv’s Boss. He’s always on a plane and loves birds.

  39. Hey Marv, I want to save some money on research.

  40. NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.

  41. TIME-AWARE RESEARCH IS THE BEST

  42. None
  43. I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE

  44. INVOLVED OBSERVERS Easier to collaborate in real time.

  45. TITLE TEXT GEOGRAPHY

  46. WHEN DOES REMOTE LOSE?

  47. None
  48. Hey Marv, I want feedback on our MMO video game

    from 13yo English speaking kids in France & Latin America with webcams.
  49. I guess, Boss. Might be hard to do with remote

    testing.
  50. BANDWIDTH Yes, it’s an issue.

  51. TRANSLATION It’s expensive and difficult to do live.

  52. MINORS Consent is a bear. Especially if they’re under 13.

  53. FACES Most users don’t have webcams. Webcams are a little

    invasive.
  54. YOUR BOSS

  55. SUPER DUPER SECURITY

  56. AN EPIC FAIL.

  57. None
  58. “What happens when you let go of a balloon?” HOW

    TO FIND ACTUAL ENGLISH SPEAKERS
  59. Bad Research We basically did

  60. Kids will lie for coins People mis-assessed their English levels

    Bandwidth was really a problem Kids don’t play when their parents are home
  61. DESIGNING REMOTE STUDIES

  62. Yo Marv, get me some insight on WHY people are

    bailing out of checkout.
  63. Sure, Boss. Let’s combine methods. Intercept & observe 10 users

    live, and do 10 self- moderated with usertesting.com
  64. TIME-AWARE TASKS Told to find purple pumps. Doesn’t care. Has

    to wear this for sister’s wedding this weekend. Whole different interview.
  65. HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.

  66. REAL TIME OBSERVERS

  67. None
  68. TITLE TEXT

  69. PROTOTYPES, WIREFRAMES, SKETCHES?

  70. HECK YES.

  71. None
  72. okay, plan these things Method Recruiting Logistics

  73. MODERATED must-haves Consent Tech set-up The beef De-installation Thanks &

    incentive
  74. UNMODERATED must-haves Expectation setting Instructions The beef Thanks & incentive

  75. Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current

    tasks Different: Compatible tech
  76. Logistics: What & How Provide test target Access Credentials

  77. Logistics: Who & When How to connect with the user

    (we’ll discuss in Tools) Managing time zones Session length
  78. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for

    location
  79. STAGED Incentives For Contextual Inquiry Start with $75, then at

    the end offer $25 more for environment photos Self-made diary studies
  80. None
  81. Advanced planning: hybrid studies Why use only one method? Some

    methods don’t tell you the why 3-5 remote interviews is all you need
  82. RECRUITING FOR REMOTE

  83. So Marv, how do we find people for these studies?

  84. Stop harassing me and get off that weird plane.

  85. RECRUITING OPTIONS Grab someone in the hallway Grab someone who

    isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness
  86. GOLD STANDARD Live recruiting Get as close as you can

  87. RECRUITING OPTIONS Ethnio Links or house ads to online forms

    Recruiting agencies Customer email lists Panels Craigslist/online ads
  88. USE ETHNIO WHEN Website access Some budget Quotas are relatively

    simple
  89. USE FORMS WHEN You can place a link but not

    code Low budget Quotas are relatively complex Options: Google Docs Wufoo
  90. USE AGENCIES WHEN No website access People whine about first-time

    visitors Higher budget Online recruiting failed
  91. CUSTOMER LISTS Sub-segments Ownership requirements No website access

  92. PANEL

  93. SORT OF

  94. None
  95. CRAIGSLIST When other methods don’t work No dough Takes a

    while Embarrassing to admit to your friends
  96. LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response

    (low average You Need: 10,000 uniques/day
  97. None
  98. FUCK TIME SHARE SALES PITCHES

  99. None
  100. TITLE TEXT

  101. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for

    location
  102. Screener constraints Keep it short Different from surveys Who gets

    paid?
  103. FIRE? see you at 4pm

  104. None
  105. Your best questions

  106. Marv-o - what in the world does automated/ unmoderated/ asynchronous

    research mean?
  107. Sure, boss. There is an AVALANCHE of UX TOOLS.

  108. “At least a little behavioral” No surveys. No standard web

    analytics.
  109. Since 2007

  110. Can tools make us better researchers?

  111. CATEGORIES 5

  112. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated

    Static Automated Live
  113. Tasks On Concepts For A Long Time People Talk To

    A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site
  114. I.A. & Navigation For Ad Agencies Ammunition Insight Just Better

    Analytics Still More Ammunition More Ammunition
  115. Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated

    Static Automated Live Deep Behavioral UX Insight
  116. Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics

    Automated Static Automated Live
  117. Ad Agencies Love It Conceptual For A Long Time Self-Moderated

    Moderated Web Analytics Automated Static Automated Live
  118. Jared Spool Hates It Conceptual For A Long Time Self-Moderated

    Moderated Web Analytics Automated Static Automated Live
  119. None
  120. Remote Research Toolkit UNMODERATED (AKA AUTOMATED)

  121. 3 parts Provide test target Record user behavior Capture comments

    (optional)
  122. USABILLA Clicks & Questions Heatmaps You do your own recruiting

    $49 - $199
  123. None
  124. None
  125. TITLE TEXT LOOP11 (UN-MODERATED)

  126. TITLE TEXT

  127. GOTOMEETING - $39/MO

  128. None
  129. None
  130. TITLE TEXT

  131. None
  132. None
  133. None
  134. None
  135. None
  136. None
  137. WHAT WE USE. A LOT. • Ethnio + GTM +

    iShowYouHD • Usabilla + Loop11 + Usertesting.com
  138. EXERCISE usertesting.com - code: uxlondon (1 free test)

  139. None
  140. RESULTS

  141. REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS

  142. WHAT THEY WORRY ABOUT

  143. WHO ARE YOU AND CAN I TRUST YOU?

  144. IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY

    FACE?
  145. WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?

  146. WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER

    AT THE END?
  147. WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?

  148. IS THIS GOING TO END UP ON YOUTUBE?

  149. OXYMORON? REMOTE EMPATHY

  150. Taking Notes Second monitor if possible Excel Hack (for time

    codes) Reframerapp.com
  151. How to rig your notes Leave one column (A) empty

    for notes. Set the cells of the next column (B) to record the clock time of each entry in A. =IF(A2=“”,“”,IF(B2=“”,NOW(),B2)) Set the cells of the next column (C) to record the difference between the first entry and the current entry =IF(ISERROR(B2-$B$2),“”,B2-$B$2) Fill down (Ctrl-D). Expect to use up to 120 rows for a session. Hide column B. You’ll never need to see it What the formulas look like Auto timestamp! Don’t write this down, it’s on page 104
  152. Observer Involvement IM Campfire Integrated Chat Passing Notes

  153. Recording Tools Phone Patch Cables Hardware Camtasia (Windows) iShowU HD

    (Mac) Software
  154. Phone patch set up

  155. None
  156. HYBRID STUDIES Numbers actually do lie sometimes And they don’t

    tell you the why 3-5 remote interviews is all you need
  157. So, what could go wrong?

  158. 3 points of failure Technology yours participant’s observer’s Recruiting Test

    Design
  159. Control what you can control Which is, your set up.

    You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.
  160. Prepare for what you can’t control User’s dog trips over

    power cord Internet affected by local weather User can’t install Doesn’t know how to install
  161. Mid-Session Tech Problems

  162. Live Recruiting Problems What if they gave a study and

    nobody came?
  163. Increase exposure When you’re not getting enough views When the

    wrong type of users are responding
  164. Increase incentives When it’s cheaper When you’re almost getting enough

    When segments are tough
  165. Switch methods When base traffic level isn’t high enough

  166. Script Problems Cut the BS.

  167. MOBILE REMOTE RESEARCH?

  168. None
  169. ADVANCED TECHNIQUES One to many moderation Multi-threading

  170. PORTABLE RESEARCH

  171. None
  172. BURN THE SCRIPT FIND OUT WHAT MATTERS.

  173. ADVANCED TECHNIQUES High Emotions

  174. TITLE TEXT

  175. TITLE TEXT

  176. None
  177. None
  178. None
  179. THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE

  180. DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the

    view finder, silly •Actual duct tape
  181. None
  182. Tony Fadell

  183. Treat every member of your team like an artist.

  184. Logic ≠ Intuition

  185. Logical photo by flickr.com/photos/80849382@N00/3306939463/

  186. Intuitive photo by flickr.com/photos/51035750608@N01/117683322/

  187. How do you know when something is innovative?

  188. “It’s been done before” - Bill Buxton | “Long nose

    of innovation”
  189. None
  190. None
  191. “It’s scary” - Marty Neumeier | The Designful Company

  192. remoteresear.ch/ias12 DISCOUNTS!!

  193. nate@boltpeters.com @boltron