Webvisions Barcelona Remote Research Workshop

Slide 1

Slide 1 text

Remote Research Workshop

Slide 2

Slide 2 text

Photo Credit: kaboodle.com

Slide 3

Slide 3 text

Remote Design Research Deﬁnition: Research where the participant is not sharing physical space with the researcher. In person Remote % of qualitative research done remotely* * Source: My wild guess

Slide 4

Slide 4 text

#WEBVISIONS #RR

Slide 5

Slide 5 text

Decision Theory & Adaptive Systems Group

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Misunderstood animated pedagogical agent or spawn of Satan? ” “

Slide 8

Slide 8 text

Bad Research

Slide 9

Slide 9 text

1) Academic Focus 2) Lab-based 3) False Premise

Slide 10

Slide 10 text

THAT IS WHY WE WENT REMOTE

Slide 11

Slide 11 text

Btw I’m Nate Bolt

Slide 12

Slide 12 text

12 YEARS 293 STUDIES 3,108 PARTICIPANTS (MODERATED) 118 ORGANIZATIONS 21,610 HOURS

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

“SOCIAL IMPACT OF DIGITAL TECHNOLOGY”

Slide 16

Slide 16 text

A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE

Slide 17

Slide 17 text

SF to PARIS IN TWO MINUTES

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

london to iceland in two minutes

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Let’s jump right in

Slide 22

Slide 22 text

What’s New with UX Research Tools?

Slide 23

Slide 23 text

A DISTINCTION

Slide 24

Slide 24 text

Moderated versus Automated

Slide 25

Slide 25 text

Moderated versus Automated 90% 10%

Slide 26

Slide 26 text

Moderated versus Automated Photo Credit: singularityhub.com

Slide 27

Slide 27 text

Integrated Recording That’s really awesome GoToMeeting 5.2 Moderated

Slide 28

Slide 28 text

Hugging Moderated Photo Credit: Diane Loviglio, Mozilla At Mozilla and MailChimp

Slide 29

Slide 29 text

Usabilla Automated Mobile

Slide 30

Slide 30 text

HOW TO BE graceful When All Your TECHNOLOGY BREAKS

Slide 31

Slide 31 text

Follow the buddy system

Slide 32

Slide 32 text

It’s laptop time

Slide 33

Slide 33 text

remoteresear.ch or dev.ethn.io/40762

Slide 34

Slide 34 text

remoteresear.ch/barcelona

Slide 35

Slide 35 text

(BROWSER TIME)

Slide 36

Slide 36 text

IN CASE THE INTERNET FAILS Research experience Most important thing to get out of today? Biggest fear or skepticism about remote research?

Slide 37

Slide 37 text

Thanks, Volunteer!

Slide 38

Slide 38 text

Four Takeaways 1. See moderated method & recording with GoToMeeting 2.Automated with Usabilla 3.Self-moderated tablet hugging 4.Quick incentive payment

Slide 39

Slide 39 text

Eﬀ you sales guy

Slide 40

Slide 40 text

WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results Pitfalls

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do in action movies?

Slide 44

Slide 44 text

How much practical research experience do I need to have for this workshop? ” - Many People “

Slide 45

Slide 45 text

Will this be a long lecture or an actual workshop? ” - Some Guy on Twitter “

Slide 46

Slide 46 text

Can we do contextual inquiries remotely? ” - Nick Dunlavey | @ndunlavey “

Slide 47

Slide 47 text

The physical and computing ecosystem

Slide 48

Slide 48 text

Now we do a little experiment

Slide 49

Slide 49 text

EXERCISE #1 “Giraﬀe” Binder PDF Page 4 Table groups Pick one moderator and one participant (they need to be in diﬀerent locations) 10 minutes of moderating

Slide 50

Slide 50 text

MODERATOR Instructions Your participant has a sheet of paper and that’s it Goal: ﬁnd out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.

Slide 51

Slide 51 text

It’s A RACE You only have 10 minutes to do this max.

Slide 52

Slide 52 text

Surprises?

Slide 53

Slide 53 text

Was there emotion?

Slide 54

Slide 54 text

WHEN DOES REMOTE WIN?

Slide 55

Slide 55 text

Marv. Creative director UX designer product manager developer IA co-founder.

Slide 56

Slide 56 text

Marv. Creative director UX designer product manager developer IA co-founder.

Slide 57

Slide 57 text

Marv’s Boss. He’s always on a plane and loves birds.

Slide 58

Slide 58 text

Marv’s Boss. He’s always on a plane and loves birds.

Slide 59

Slide 59 text

Hey Marv, I want to save some money on research.

Slide 60

Slide 60 text

NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.

Slide 61

Slide 61 text

TIME-AWARE RESEARCH IS THE BEST

Slide 62

Slide 62 text

I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE

Slide 63

Slide 63 text

INVOLVED OBSERVERS Easier to collaborate in real time.

Slide 64

Slide 64 text

TITLE TEXT GEOGRAPHY

Slide 65

Slide 65 text

WHEN DOES REMOTE LOSE?

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

Hey Marv, I want feedback on our MMO video game from 13yo English speaking kids in France & Latin America with webcams.

Slide 68

Slide 68 text

I guess, Boss. Might be hard to do with remote testing.

Slide 69

Slide 69 text

BANDWIDTH Yes, it’s an issue.

Slide 70

Slide 70 text

TRANSLATION It’s expensive and diﬃcult to do live.

Slide 71

Slide 71 text

MINORS Consent is a bear. Especially if they’re under 13.

Slide 72

Slide 72 text

FACES Most users don’t have webcams. Webcams are a little invasive.

Slide 73

Slide 73 text

YOUR BOSS

Slide 74

Slide 74 text

SUPER DUPER SECURITY

Slide 75

Slide 75 text

AN EPIC FAIL.

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

Bad Research We basically did

Slide 79

Slide 79 text

Kids will lie for coins People mis-assessed their English levels Bandwidth was really a problem Kids don’t play when their parents are home

Slide 80

Slide 80 text

“ What happens when you let go of a balloon? ” HOW TO FIND ACTUAL SPEAKERS OF YOUR LANGUAGE IN ANY COUNTRY: “ Qu'est-ce qui se passe quand vous lâchez un ballon? ”

Slide 81

Slide 81 text

DESIGNING REMOTE STUDIES

Slide 82

Slide 82 text

Yo Marv, get me some insight on WHY people are bailing out of checkout.

Slide 83

Slide 83 text

Sure, Boss. Let’s combine methods. Intercept & observe 10 users live, and do 10 self- moderated with usertesting.com

Slide 84

Slide 84 text

TIME-AWARE TASKS Told to ﬁnd purple pumps. Doesn’t care. Has to wear this for sister’s wedding this weekend. Whole diﬀerent interview.

Slide 85

Slide 85 text

HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.

Slide 86

Slide 86 text

REAL TIME OBSERVERS

Slide 87

Slide 87 text

No content

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

TITLE TEXT

Slide 90

Slide 90 text

PROTOTYPES, WIREFRAMES, SKETCHES?

Slide 91

Slide 91 text

HECK YES.

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

RECRUITING FOR REMOTE

Slide 94

Slide 94 text

RECRUITING OPTIONS Ethnio Links or house ads to online forms Recruiting agencies Customer email lists Panels Craigslist/online ads

Slide 95

Slide 95 text

No content

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

RECRUITING OPTIONS Grab someone in the hallway Grab someone who isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness

Slide 99

Slide 99 text

SITE ISN’T AN OPTION GEO IP MEMBERS ONLY

Slide 100

Slide 100 text

GOLD STANDARD Live recruiting Get as close as you can

Slide 101

Slide 101 text

USE ETHNIO WHEN Website access Some budget Quotas are relatively simple

Slide 102

Slide 102 text

USE FORMS WHEN You can place a link but not code Low budget Quotas are relatively complex Options: Google Docs Wufoo

Slide 103

Slide 103 text

USE AGENCIES WHEN No website access People whine about ﬁrst-time visitors Higher budget Online recruiting failed

Slide 104

Slide 104 text

CUSTOMER LISTS Sub-segments Ownership requirements No website access

Slide 105

Slide 105 text

PANEL

Slide 106

Slide 106 text

SORT OF

Slide 107

Slide 107 text

No content

Slide 108

Slide 108 text

CRAIGSLIST When other methods don’t work No dough Takes a while Embarrassing to admit to your friends

Slide 109

Slide 109 text

LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response (low average You Need: 10,000 uniques/day

Slide 110

Slide 110 text

TITLE TEXT

Slide 111

Slide 111 text

remoteresear.ch/samples “Online Consent”

Slide 112

Slide 112 text

Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location

Slide 113

Slide 113 text

Screener constraints Keep it short Diﬀerent from surveys Who gets paid?

Slide 114

Slide 114 text

MODERATED must-haves

Slide 115

Slide 115 text

MODERATED must-haves Consent

Slide 116

Slide 116 text

MODERATED must-haves Consent Tech set-up

Slide 117

Slide 117 text

MODERATED must-haves Consent Tech set-up The beef

Slide 118

Slide 118 text

MODERATED must-haves Consent Tech set-up The beef De-installation

Slide 119

Slide 119 text

MODERATED must-haves Consent Tech set-up The beef De-installation Thanks & incentive

Slide 120

Slide 120 text

AUTOMATED must-haves

Slide 121

Slide 121 text

AUTOMATED must-haves Expectation setting

Slide 122

Slide 122 text

AUTOMATED must-haves Expectation setting Instructions

Slide 123

Slide 123 text

AUTOMATED must-haves Expectation setting Instructions The beef

Slide 124

Slide 124 text

AUTOMATED must-haves Expectation setting Instructions The beef Thanks & incentive

Slide 125

Slide 125 text

Recruiting

Slide 126

Slide 126 text

Recruiting Same: Good “talkers” Same: Demographic ﬁts

Slide 127

Slide 127 text

Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current tasks Different: Compatible tech

Slide 128

Slide 128 text

Logistics: What & How Provide test target Access Credentials

Slide 129

Slide 129 text

Logistics: Who & When How to connect with the user Managing time zones Session length

Slide 130

Slide 130 text

Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location

Slide 131

Slide 131 text

STAGED Incentives For Contextual Inquiry Start with $75, then at the end oﬀer $25 more for environment photos Self-made diary studies

Slide 132

Slide 132 text

No content

Slide 133

Slide 133 text

Advanced planning: hybrid studies Why use only one method? Some methods don’t tell you the why 3-5 remote interviews is all you need

Slide 134

Slide 134 text

EXERCISE #2 okay, draft a screener

Slide 135

Slide 135 text

EXERCISE #2 Binder PDF Page 6 Use ethn.io or dev.ethn.io (beta) or wufoo Under 8 Questions okay, draft a screener

Slide 136

Slide 136 text

EXERCISE #2 Binder PDF Page 6 Use ethn.io or dev.ethn.io (beta) or wufoo Under 8 Questions Advanced Bonus: create an automated and moderated screener okay, draft a screener

Slide 137

Slide 137 text

No content

Slide 138

Slide 138 text

Marv-o - what tools should we pick? I mean, tell me to use for pete’s ake

Slide 139

Slide 139 text

Sure, boss. There is an AVALANCHE of UX TOOLS.

Slide 140

Slide 140 text

“At least a little behavioral” No surveys. No standard web analytics.

Slide 141

Slide 141 text

No content

Slide 142

Slide 142 text

Since 2007

Slide 143

Slide 143 text

Can tools make us better researchers?

Slide 144

Slide 144 text

CATEGORIES 5

Slide 145

Slide 145 text

Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 146

Slide 146 text

Tasks On Concepts For A Long Time People Talk To A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site

Slide 147

Slide 147 text

I.A. & Navigation For Ad Agencies Ammunition Insight Just Better Analytics Still More Ammunition More Ammunition

Slide 148

Slide 148 text

Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live Deep Behavioral UX Insight

Slide 149

Slide 149 text

Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 150

Slide 150 text

Ad Agencies Love It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 151

Slide 151 text

Jared Spool Hates It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 152

Slide 152 text

Jared Spool Hates It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 153

Slide 153 text

No content

Slide 154

Slide 154 text

Mobile Conceptual For A Long Time Self-Moderated Moderated Automated Static Automated Live }

Slide 155

Slide 155 text

Remote Research Toolkit UNMODERATED (AKA AUTOMATED)

Slide 156

Slide 156 text

3 parts Provide test target Record user behavior Capture comments (optional)

Slide 157

Slide 157 text

USABILLA Clicks & Questions Heatmaps You do your own recruiting $49 - $199

Slide 158

Slide 158 text

No content

Slide 159

Slide 159 text

No content

Slide 160

Slide 160 text

No content

Slide 161

Slide 161 text

No content

Slide 162

Slide 162 text

TITLE TEXT LOOP11 (UN-MODERATED)

Slide 163

Slide 163 text

TITLE TEXT

Slide 164

Slide 164 text

GOTOMEETING - $39/MO

Slide 165

Slide 165 text

No content

Slide 166

Slide 166 text

No content

Slide 167

Slide 167 text

TITLE TEXT

Slide 168

Slide 168 text

No content

Slide 169

Slide 169 text

No content

Slide 170

Slide 170 text

No content

Slide 171

Slide 171 text

No content

Slide 172

Slide 172 text

No content

Slide 173

Slide 173 text

No content

Slide 174

Slide 174 text

WHAT WE USE. A LOT. • Ethnio + GTM • Usabilla + Usertesting.com • Silverback (Mac) & Camtasia (PC) • Guerrilla mobile

Slide 175

Slide 175 text

No content

Slide 176

Slide 176 text

RESULTS

Slide 177

Slide 177 text

REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS

Slide 178

Slide 178 text

WHAT THEY WORRY ABOUT

Slide 179

Slide 179 text

WHO ARE YOU AND CAN I TRUST YOU?

Slide 180

Slide 180 text

IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY FACE?

Slide 181

Slide 181 text

WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?

Slide 182

Slide 182 text

WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER AT THE END?

Slide 183

Slide 183 text

WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?

Slide 184

Slide 184 text

IS THIS GOING TO END UP ON YOUTUBE?

Slide 185

Slide 185 text

OXYMORON? REMOTE EMPATHY

Slide 186

Slide 186 text

Taking Notes Second monitor if possible Excel Hack (remoteresear.ch/samples)

Slide 187

Slide 187 text

Observer Involvement IM Campﬁre (!) Integrated Chat Passing Notes

Slide 188

Slide 188 text

HYBRID STUDIES Numbers actually do lie sometimes And they don’t tell you the why 3-5 remote interviews is all you need

Slide 189

Slide 189 text

So, what could go wrong?

Slide 190

Slide 190 text

3 points of failure Technology yours participant’s observer’s Recruiting Test Design

Slide 191

Slide 191 text

Control what you can control Which is, your set up. You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.

Slide 192

Slide 192 text

Prepare for what you can’t control User’s dog trips over power cord Internet aﬀected by local weather User can’t install Doesn’t know how to install

Slide 193

Slide 193 text

Mid-Session Tech Problems

Slide 194

Slide 194 text

Live Recruiting Problems What if they gave a study and nobody came?

Slide 195

Slide 195 text

Increase exposure When you’re not getting enough views When the wrong type of users are responding

Slide 196

Slide 196 text

Increase incentives When it’s cheaper When you’re almost getting enough When segments are tough

Slide 197

Slide 197 text

Switch methods When base traﬃc level isn’t high enough

Slide 198

Slide 198 text

Script Problems Cut the BS.

Slide 199

Slide 199 text

ADVANCED TECHNIQUES One to many moderation Multi-threading

Slide 200

Slide 200 text

PORTABLE RESEARCH

Slide 201

Slide 201 text

No content

Slide 202

Slide 202 text

BURN THE SCRIPT FIND OUT WHAT MATTERS.

Slide 203

Slide 203 text

ADVANCED TECHNIQUES High Emotions

Slide 204

Slide 204 text

TITLE TEXT

Slide 205

Slide 205 text

No content

Slide 206

Slide 206 text

No content

Slide 207

Slide 207 text

No content

Slide 208

Slide 208 text

THE FUTURE

Slide 209

Slide 209 text

THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE

Slide 210

Slide 210 text

DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the view ﬁnder, silly •Actual duct tape

Slide 211

Slide 211 text

No content

Slide 212

Slide 212 text

nate@boltron.com @boltron