Remote Research Workshop

Slide 1

Slide 1 text

Photo Credit: Werner Schnell on Flickr

Slide 2

Slide 2 text

REMOTE RESEARCH Deﬁnition: Research where the participant is not sharing physical space with the researcher. In person Remote % of qualitative research done remotely* *Source: BP’s best guess

Slide 3

Slide 3 text

#IAS12 #RR

Slide 4

Slide 4 text

Bayesian methods Animated pedagogical agency 1995 Infer when users needed help Decision Theory & Adaptive Systems Group

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

“Misunderstood animated pedagogical agent or spawn of Satan?”

Slide 7

Slide 7 text

Bad Research

Slide 8

Slide 8 text

1) Academic Focus 2) Lab-based 3) False Premise

Slide 9

Slide 9 text

THAT IS WHY WE WENT REMOTE

Slide 10

Slide 10 text

Let’s jump right in.

Slide 11

Slide 11 text

HOW TO BE graceful When All Your TECHNOLOGY BREAKS

Slide 12

Slide 12 text

Follow the buddy system

Slide 13

Slide 13 text

It’s laptop time

Slide 14

Slide 14 text

remoteresear.ch

Slide 15

Slide 15 text

IN CASE THE INTERNET FAILS Research experience Most important thing to get out of today? Biggest fear or skepticism about remote research?

Slide 16

Slide 16 text

“How much practical research experience do I need to have for this workshop?” - Many People

Slide 17

Slide 17 text

10 YEARS 244 STUDIES 2,915 PARTICIPANTS (MODERATED) 95 ORGANIZATIONS 19,610 HOURS

Slide 18

Slide 18 text

“SOCIAL IMPACT OF DIGITAL TECHNOLOGY”

Slide 19

Slide 19 text

OMG PLEASE BUY THIS BOOK

Slide 20

Slide 20 text

60% PROTOTYPE RESEARCH

Slide 21

Slide 21 text

A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

“Can we do contextual inquiries remotely?” - Nick Dunlavey | @ndunlavey

Slide 25

Slide 25 text

The physical and computing ecosystem

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do in action movies?

Slide 28

Slide 28 text

WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results Pitfalls

Slide 29

Slide 29 text

Now we do a little experiment

Slide 30

Slide 30 text

EXERCISE JUMP IN THE POOL Table groups Pick one moderator and one participant (they need to be in diﬀerent locations) 10 minutes of moderating

Slide 31

Slide 31 text

MODERATOR Instructions Your participant has a sheet of paper and that’s it Goal: ﬁnd out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.

Slide 32

Slide 32 text

It’s A RACE First participant to make a beautiful oragami giraﬀe is the winner

Slide 33

Slide 33 text

Up to 10 Minutes

Slide 34

Slide 34 text

Surprises?

Slide 35

Slide 35 text

Was there emotion?

Slide 36

Slide 36 text

WHEN DOES REMOTE WIN?

Slide 37

Slide 37 text

Marv. Creative director UX designer product manager developer IA co-founder.

Slide 38

Slide 38 text

Marv’s Boss. He’s always on a plane and loves birds.

Slide 39

Slide 39 text

Hey Marv, I want to save some money on research.

Slide 40

Slide 40 text

NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.

Slide 41

Slide 41 text

TIME-AWARE RESEARCH IS THE BEST

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE

Slide 44

Slide 44 text

INVOLVED OBSERVERS Easier to collaborate in real time.

Slide 45

Slide 45 text

TITLE TEXT GEOGRAPHY

Slide 46

Slide 46 text

WHEN DOES REMOTE LOSE?

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Hey Marv, I want feedback on our MMO video game from 13yo English speaking kids in France & Latin America with webcams.

Slide 49

Slide 49 text

I guess, Boss. Might be hard to do with remote testing.

Slide 50

Slide 50 text

BANDWIDTH Yes, it’s an issue.

Slide 51

Slide 51 text

TRANSLATION It’s expensive and diﬃcult to do live.

Slide 52

Slide 52 text

MINORS Consent is a bear. Especially if they’re under 13.

Slide 53

Slide 53 text

FACES Most users don’t have webcams. Webcams are a little invasive.

Slide 54

Slide 54 text

YOUR BOSS

Slide 55

Slide 55 text

SUPER DUPER SECURITY

Slide 56

Slide 56 text

AN EPIC FAIL.

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

“What happens when you let go of a balloon?” HOW TO FIND ACTUAL ENGLISH SPEAKERS

Slide 59

Slide 59 text

Bad Research We basically did

Slide 60

Slide 60 text

Kids will lie for coins People mis-assessed their English levels Bandwidth was really a problem Kids don’t play when their parents are home

Slide 61

Slide 61 text

DESIGNING REMOTE STUDIES

Slide 62

Slide 62 text

Yo Marv, get me some insight on WHY people are bailing out of checkout.

Slide 63

Slide 63 text

Sure, Boss. Let’s combine methods. Intercept & observe 10 users live, and do 10 self- moderated with usertesting.com

Slide 64

Slide 64 text

TIME-AWARE TASKS Told to ﬁnd purple pumps. Doesn’t care. Has to wear this for sister’s wedding this weekend. Whole diﬀerent interview.

Slide 65

Slide 65 text

HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.

Slide 66

Slide 66 text

REAL TIME OBSERVERS

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

TITLE TEXT

Slide 69

Slide 69 text

PROTOTYPES, WIREFRAMES, SKETCHES?

Slide 70

Slide 70 text

HECK YES.

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

okay, plan these things Method Recruiting Logistics

Slide 73

Slide 73 text

MODERATED must-haves Consent Tech set-up The beef De-installation Thanks & incentive

Slide 74

Slide 74 text

UNMODERATED must-haves Expectation setting Instructions The beef Thanks & incentive

Slide 75

Slide 75 text

Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current tasks Different: Compatible tech

Slide 76

Slide 76 text

Logistics: What & How Provide test target Access Credentials

Slide 77

Slide 77 text

Logistics: Who & When How to connect with the user (we’ll discuss in Tools) Managing time zones Session length

Slide 78

Slide 78 text

Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location

Slide 79

Slide 79 text

STAGED Incentives For Contextual Inquiry Start with $75, then at the end oﬀer $25 more for environment photos Self-made diary studies

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

Advanced planning: hybrid studies Why use only one method? Some methods don’t tell you the why 3-5 remote interviews is all you need

Slide 82

Slide 82 text

RECRUITING FOR REMOTE

Slide 83

Slide 83 text

So Marv, how do we ﬁnd people for these studies?

Slide 84

Slide 84 text

Stop harassing me and get oﬀ that weird plane.

Slide 85

Slide 85 text

RECRUITING OPTIONS Grab someone in the hallway Grab someone who isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness

Slide 86

Slide 86 text

GOLD STANDARD Live recruiting Get as close as you can

Slide 87

Slide 87 text

RECRUITING OPTIONS Ethnio Links or house ads to online forms Recruiting agencies Customer email lists Panels Craigslist/online ads

Slide 88

Slide 88 text

USE ETHNIO WHEN Website access Some budget Quotas are relatively simple

Slide 89

Slide 89 text

USE FORMS WHEN You can place a link but not code Low budget Quotas are relatively complex Options: Google Docs Wufoo

Slide 90

Slide 90 text

USE AGENCIES WHEN No website access People whine about ﬁrst-time visitors Higher budget Online recruiting failed

Slide 91

Slide 91 text

CUSTOMER LISTS Sub-segments Ownership requirements No website access

Slide 92

Slide 92 text

PANEL

Slide 93

Slide 93 text

SORT OF

Slide 94

Slide 94 text

No content

Slide 95

Slide 95 text

CRAIGSLIST When other methods don’t work No dough Takes a while Embarrassing to admit to your friends

Slide 96

Slide 96 text

LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response (low average You Need: 10,000 uniques/day

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

FUCK TIME SHARE SALES PITCHES

Slide 99

Slide 99 text

No content

Slide 100

Slide 100 text

TITLE TEXT

Slide 101

Slide 101 text

Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location

Slide 102

Slide 102 text

Screener constraints Keep it short Diﬀerent from surveys Who gets paid?

Slide 103

Slide 103 text

FIRE? see you at 4pm

Slide 104

Slide 104 text

No content

Slide 105

Slide 105 text

Your best questions

Slide 106

Slide 106 text

Marv-o - what in the world does automated/ unmoderated/ asynchronous research mean?

Slide 107

Slide 107 text

Sure, boss. There is an AVALANCHE of UX TOOLS.

Slide 108

Slide 108 text

“At least a little behavioral” No surveys. No standard web analytics.

Slide 109

Slide 109 text

Since 2007

Slide 110

Slide 110 text

Can tools make us better researchers?

Slide 111

Slide 111 text

CATEGORIES 5

Slide 112

Slide 112 text

Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 113

Slide 113 text

Tasks On Concepts For A Long Time People Talk To A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site

Slide 114

Slide 114 text

I.A. & Navigation For Ad Agencies Ammunition Insight Just Better Analytics Still More Ammunition More Ammunition

Slide 115

Slide 115 text

Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live Deep Behavioral UX Insight

Slide 116

Slide 116 text

Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 117

Slide 117 text

Ad Agencies Love It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 118

Slide 118 text

Jared Spool Hates It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 119

Slide 119 text

No content

Slide 120

Slide 120 text

Remote Research Toolkit UNMODERATED (AKA AUTOMATED)

Slide 121

Slide 121 text

3 parts Provide test target Record user behavior Capture comments (optional)

Slide 122

Slide 122 text

USABILLA Clicks & Questions Heatmaps You do your own recruiting $49 - $199

Slide 123

Slide 123 text

No content

Slide 124

Slide 124 text

No content

Slide 125

Slide 125 text

TITLE TEXT LOOP11 (UN-MODERATED)

Slide 126

Slide 126 text

TITLE TEXT

Slide 127

Slide 127 text

GOTOMEETING - $39/MO

Slide 128

Slide 128 text

No content

Slide 129

Slide 129 text

No content

Slide 130

Slide 130 text

TITLE TEXT

Slide 131

Slide 131 text

No content

Slide 132

Slide 132 text

No content

Slide 133

Slide 133 text

No content

Slide 134

Slide 134 text

No content

Slide 135

Slide 135 text

No content

Slide 136

Slide 136 text

No content

Slide 137

Slide 137 text

WHAT WE USE. A LOT. • Ethnio + GTM + iShowYouHD • Usabilla + Loop11 + Usertesting.com

Slide 138

Slide 138 text

EXERCISE usertesting.com - code: uxlondon (1 free test)

Slide 139

Slide 139 text

No content

Slide 140

Slide 140 text

RESULTS

Slide 141

Slide 141 text

REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS

Slide 142

Slide 142 text

WHAT THEY WORRY ABOUT

Slide 143

Slide 143 text

WHO ARE YOU AND CAN I TRUST YOU?

Slide 144

Slide 144 text

IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY FACE?

Slide 145

Slide 145 text

WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?

Slide 146

Slide 146 text

WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER AT THE END?

Slide 147

Slide 147 text

WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?

Slide 148

Slide 148 text

IS THIS GOING TO END UP ON YOUTUBE?

Slide 149

Slide 149 text

OXYMORON? REMOTE EMPATHY

Slide 150

Slide 150 text

Taking Notes Second monitor if possible Excel Hack (for time codes) Reframerapp.com

Slide 151

Slide 151 text

How to rig your notes Leave one column (A) empty for notes. Set the cells of the next column (B) to record the clock time of each entry in A. =IF(A2=“”,“”,IF(B2=“”,NOW(),B2)) Set the cells of the next column (C) to record the diﬀerence between the ﬁrst entry and the current entry =IF(ISERROR(B2-$B$2),“”,B2-$B$2) Fill down (Ctrl-D). Expect to use up to 120 rows for a session. Hide column B. You’ll never need to see it What the formulas look like Auto timestamp! Don’t write this down, it’s on page 104

Slide 152

Slide 152 text

Observer Involvement IM Campﬁre Integrated Chat Passing Notes

Slide 153

Slide 153 text

Recording Tools Phone Patch Cables Hardware Camtasia (Windows) iShowU HD (Mac) Software

Slide 154

Slide 154 text

Phone patch set up

Slide 155

Slide 155 text

No content

Slide 156

Slide 156 text

HYBRID STUDIES Numbers actually do lie sometimes And they don’t tell you the why 3-5 remote interviews is all you need

Slide 157

Slide 157 text

So, what could go wrong?

Slide 158

Slide 158 text

3 points of failure Technology yours participant’s observer’s Recruiting Test Design

Slide 159

Slide 159 text

Control what you can control Which is, your set up. You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.

Slide 160

Slide 160 text

Prepare for what you can’t control User’s dog trips over power cord Internet aﬀected by local weather User can’t install Doesn’t know how to install

Slide 161

Slide 161 text

Mid-Session Tech Problems

Slide 162

Slide 162 text

Live Recruiting Problems What if they gave a study and nobody came?

Slide 163

Slide 163 text

Increase exposure When you’re not getting enough views When the wrong type of users are responding

Slide 164

Slide 164 text

Increase incentives When it’s cheaper When you’re almost getting enough When segments are tough

Slide 165

Slide 165 text

Switch methods When base traﬃc level isn’t high enough

Slide 166

Slide 166 text

Script Problems Cut the BS.

Slide 167

Slide 167 text

MOBILE REMOTE RESEARCH?

Slide 168

Slide 168 text

No content

Slide 169

Slide 169 text

ADVANCED TECHNIQUES One to many moderation Multi-threading

Slide 170

Slide 170 text

PORTABLE RESEARCH

Slide 171

Slide 171 text

No content

Slide 172

Slide 172 text

BURN THE SCRIPT FIND OUT WHAT MATTERS.

Slide 173

Slide 173 text

ADVANCED TECHNIQUES High Emotions

Slide 174

Slide 174 text

TITLE TEXT

Slide 175

Slide 175 text

TITLE TEXT

Slide 176

Slide 176 text

No content

Slide 177

Slide 177 text

No content

Slide 178

Slide 178 text

No content

Slide 179

Slide 179 text

THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE

Slide 180

Slide 180 text

DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the view ﬁnder, silly •Actual duct tape

Slide 181

Slide 181 text

No content

Slide 182

Slide 182 text

Tony Fadell

Slide 183

Slide 183 text

Treat every member of your team like an artist.

Slide 184

Slide 184 text

Logic ≠ Intuition

Slide 185

Slide 185 text

Logical photo by ﬂickr.com/photos/80849382@N00/3306939463/

Slide 186

Slide 186 text

Intuitive photo by ﬂickr.com/photos/51035750608@N01/117683322/

Slide 187

Slide 187 text

How do you know when something is innovative?

Slide 188

Slide 188 text

“It’s been done before” - Bill Buxton | “Long nose of innovation”

Slide 189

Slide 189 text

No content

Slide 190

Slide 190 text

No content

Slide 191

Slide 191 text

“It’s scary” - Marty Neumeier | The Designful Company

Slide 192

Slide 192 text

remoteresear.ch/ias12 DISCOUNTS!!

Slide 193

Slide 193 text

[email protected] @boltron