UX Lisbon Remote Research Workshop

Slide 1

Slide 1 text

Remote Research Workshop

Slide 2

Slide 2 text

Photo Credit: recados-orkut.net

Slide 3

Slide 3 text

Remote Design Research Deﬁnition: Research where the participant is not sharing physical space with the researcher. In person Remote % of qualitative research done remotely* * Source: My wild guess

Slide 4

Slide 4 text

#UXLX #RR

Slide 5

Slide 5 text

Bayesian methods Animated pedagogical agency 1995 Infer when users needed help Decision Theory & Adaptive Systems Group

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Misunderstood animated pedagogical agent or spawn of Satan? ” “

Slide 8

Slide 8 text

Bad Research

Slide 9

Slide 9 text

1) Academic Focus 2) Lab-based 3) False Premise

Slide 10

Slide 10 text

THAT IS WHY WE WENT REMOTE

Slide 11

Slide 11 text

Btw I’m Nate Bolt

Slide 12

Slide 12 text

12 YEARS 293 STUDIES 3,108 PARTICIPANTS (MODERATED) 118 ORGANIZATIONS 21,610 HOURS

Slide 13

Slide 13 text

“SOCIAL IMPACT OF DIGITAL TECHNOLOGY”

Slide 14

Slide 14 text

A DINOSAUR FAMILY EXPLAINS INFORMATION ARCHITECTURE

Slide 15

Slide 15 text

SF to PARIS IN TWO MINUTES

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

jumboltron.com

Slide 18

Slide 18 text

OMG PLEASE BUY THIS BOOK

Slide 19

Slide 19 text

Let’s jump right in

Slide 20

Slide 20 text

What’s New with UX Research Tools?

Slide 21

Slide 21 text

BUT FIRST

Slide 22

Slide 22 text

Moderated versus Automated

Slide 23

Slide 23 text

Moderated versus Automated 90% 10%

Slide 24

Slide 24 text

Moderated versus Automated Photo Credit: singularityhub.com

Slide 25

Slide 25 text

Integrated Recording That’s really awesome GoToMeeting 5.2 Moderated

Slide 26

Slide 26 text

Hugging Moderated Photo Credit: Diane Loviglio, Mozilla At Mozilla and MailChimp

Slide 27

Slide 27 text

Usabilla Automated Mobile

Slide 28

Slide 28 text

HOW TO BE graceful When All Your TECHNOLOGY BREAKS

Slide 29

Slide 29 text

Follow the buddy system

Slide 30

Slide 30 text

It’s laptop time

Slide 31

Slide 31 text

remoteresear.ch

Slide 32

Slide 32 text

remoteresear.ch/lisbon

Slide 33

Slide 33 text

(BROWSER TIME)

Slide 34

Slide 34 text

IN CASE THE INTERNET FAILS Research experience Most important thing to get out of today? Biggest fear or skepticism about remote research?

Slide 35

Slide 35 text

Thanks, Volunteer!

Slide 36

Slide 36 text

Four Takeaways 1. See moderated method & recording with GoToMeeting 2.Automated with Usabilla 3.Self-moderated tablet hugging 4.Quick incentive payment

Slide 37

Slide 37 text

Eﬀ you sales guy

Slide 38

Slide 38 text

WHAT WE’LL COVER TODAY Recruiting Moderated Contextual Inquiry Automated Results Pitfalls

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Explode SOME THOUGHTS ON WEB ANALYTICS What do helicopters do in action movies?

Slide 41

Slide 41 text

How much practical research experience do I need to have for this workshop? ” - Many People “

Slide 42

Slide 42 text

Will this be a long lecture or an actual workshop? ” - Some Guy on Twitter “

Slide 43

Slide 43 text

Can we do contextual inquiries remotely? ” - Nick Dunlavey | @ndunlavey “

Slide 44

Slide 44 text

The physical and computing ecosystem

Slide 45

Slide 45 text

Now we do a little experiment

Slide 46

Slide 46 text

EXERCISE #1 “Giraﬀe” Binder PDF Page 4 Table groups Pick one moderator and one participant (they need to be in diﬀerent locations) 10 minutes of moderating

Slide 47

Slide 47 text

remoteresear.ch/lisbon “The Workshop Binder”

Slide 48

Slide 48 text

MODERATOR Instructions Your participant has a sheet of paper and that’s it Goal: ﬁnd out what kind of physical task instructions online Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.

Slide 49

Slide 49 text

It’s A RACE You only have 10 minutes to do this max.

Slide 50

Slide 50 text

Surprises?

Slide 51

Slide 51 text

Was there emotion?

Slide 52

Slide 52 text

WHEN DOES REMOTE WIN?

Slide 53

Slide 53 text

Marv. Creative director UX designer product manager developer IA co-founder.

Slide 54

Slide 54 text

Marv’s Boss. He’s always on a plane and loves birds.

Slide 55

Slide 55 text

Hey Marv, I want to save some money on research.

Slide 56

Slide 56 text

NATIVE ENVIRONMENT CHEAP ETHNOGRAPHY, NOT CHEAP LAB.

Slide 57

Slide 57 text

TIME-AWARE RESEARCH IS THE BEST

Slide 58

Slide 58 text

I’LL TRADE YOU TIME FOR PHYSICAL PRESENCE

Slide 59

Slide 59 text

INVOLVED OBSERVERS Easier to collaborate in real time.

Slide 60

Slide 60 text

TITLE TEXT GEOGRAPHY

Slide 61

Slide 61 text

WHEN DOES REMOTE LOSE?

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

Hey Marv, I want feedback on our MMO video game from 13yo English speaking kids in France & Latin America with webcams.

Slide 64

Slide 64 text

I guess, Boss. Might be hard to do with remote testing.

Slide 65

Slide 65 text

BANDWIDTH Yes, it’s an issue.

Slide 66

Slide 66 text

TRANSLATION It’s expensive and diﬃcult to do live.

Slide 67

Slide 67 text

MINORS Consent is a bear. Especially if they’re under 13.

Slide 68

Slide 68 text

FACES Most users don’t have webcams. Webcams are a little invasive.

Slide 69

Slide 69 text

YOUR BOSS

Slide 70

Slide 70 text

SUPER DUPER SECURITY

Slide 71

Slide 71 text

AN EPIC FAIL.

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

Bad Research We basically did

Slide 74

Slide 74 text

Kids will lie for coins People mis-assessed their English levels Bandwidth was really a problem Kids don’t play when their parents are home

Slide 75

Slide 75 text

“ What happens when you let go of a balloon? ” HOW TO FIND ACTUAL SPEAKERS OF YOUR LANGUAGE IN ANY COUNTRY: “ Qu'est-ce qui se passe quand vous lâchez un ballon? ”

Slide 76

Slide 76 text

DESIGNING REMOTE STUDIES

Slide 77

Slide 77 text

Yo Marv, get me some insight on WHY people are bailing out of checkout.

Slide 78

Slide 78 text

Sure, Boss. Let’s combine methods. Intercept & observe 10 users live, and do 10 self- moderated with usertesting.com

Slide 79

Slide 79 text

TIME-AWARE TASKS Told to ﬁnd purple pumps. Doesn’t care. Has to wear this for sister’s wedding this weekend. Whole diﬀerent interview.

Slide 80

Slide 80 text

HEART & SOUL OF REMOTE RESEARCH Not scheduling participants.

Slide 81

Slide 81 text

REAL TIME OBSERVERS

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

TITLE TEXT

Slide 84

Slide 84 text

PROTOTYPES, WIREFRAMES, SKETCHES?

Slide 85

Slide 85 text

HECK YES.

Slide 86

Slide 86 text

No content

Slide 87

Slide 87 text

RECRUITING FOR REMOTE

Slide 88

Slide 88 text

RECRUITING OPTIONS Ethnio Links or house ads to online forms Recruiting agencies Customer email lists Panels Craigslist/online ads

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

RECRUITING OPTIONS Grab someone in the hallway Grab someone who isn’t directly involved in your project IM your friends for a quick look Panels and recruiting agencies Real customers from marketing lists Site intercepts realness

Slide 91

Slide 91 text

SITE ISN’T AN OPTION GEO IP MEMBERS ONLY

Slide 92

Slide 92 text

GOLD STANDARD Live recruiting Get as close as you can

Slide 93

Slide 93 text

USE ETHNIO WHEN Website access Some budget Quotas are relatively simple

Slide 94

Slide 94 text

USE FORMS WHEN You can place a link but not code Low budget Quotas are relatively complex Options: Google Docs Wufoo

Slide 95

Slide 95 text

USE AGENCIES WHEN No website access People whine about ﬁrst-time visitors Higher budget Online recruiting failed

Slide 96

Slide 96 text

CUSTOMER LISTS Sub-segments Ownership requirements No website access

Slide 97

Slide 97 text

PANEL

Slide 98

Slide 98 text

SORT OF

Slide 99

Slide 99 text

No content

Slide 100

Slide 100 text

CRAIGSLIST When other methods don’t work No dough Takes a while Embarrassing to admit to your friends

Slide 101

Slide 101 text

LIVE RECRUITING MATH Target: 6 recruits/hour Allow for: 1% response (low average You Need: 10,000 uniques/day

Slide 102

Slide 102 text

TITLE TEXT

Slide 103

Slide 103 text

remoteresear.ch/samples “Online Consent”

Slide 104

Slide 104 text

Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location

Slide 105

Slide 105 text

Screener constraints Keep it short Diﬀerent from surveys Who gets paid?

Slide 106

Slide 106 text

MODERATED must-haves Consent Tech set-up The beef De-installation Thanks & incentive

Slide 107

Slide 107 text

AUTOMATED must-haves Expectation setting Instructions The beef Thanks & incentive

Slide 108

Slide 108 text

Recruiting Same: Good “talkers” Same: Demographic fits Different: Relevant current tasks Different: Compatible tech

Slide 109

Slide 109 text

Logistics: What & How Provide test target Access Credentials

Slide 110

Slide 110 text

Logistics: Who & When How to connect with the user Managing time zones Session length

Slide 111

Slide 111 text

Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location

Slide 112

Slide 112 text

STAGED Incentives For Contextual Inquiry Start with $75, then at the end oﬀer $25 more for environment photos Self-made diary studies

Slide 113

Slide 113 text

No content

Slide 114

Slide 114 text

Advanced planning: hybrid studies Why use only one method? Some methods don’t tell you the why 3-5 remote interviews is all you need

Slide 115

Slide 115 text

EXERCISE #2 Binder PDF Page 6 Use ethn.io or dev.ethn.io (beta) or wufoo Under 8 Questions Advanced Bonus: create an automated and moderated screener okay, draft a screener

Slide 116

Slide 116 text

remoteresear.ch/lisbon “The Workshop Binder”

Slide 117

Slide 117 text

Marv-o - what in the world does automated/ unmoderated/ asynchronous research mean?

Slide 118

Slide 118 text

Sure, boss. There is an AVALANCHE of UX TOOLS.

Slide 119

Slide 119 text

“At least a little behavioral” No surveys. No standard web analytics.

Slide 120

Slide 120 text

Since 2007

Slide 121

Slide 121 text

Can tools make us better researchers?

Slide 122

Slide 122 text

CATEGORIES 5

Slide 123

Slide 123 text

Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 124

Slide 124 text

Tasks On Concepts For A Long Time People Talk To A Machine Talk To People Fancy-Ass Analytics Tasks On Images Tasks On A Live Site

Slide 125

Slide 125 text

I.A. & Navigation For Ad Agencies Ammunition Insight Just Better Analytics Still More Ammunition More Ammunition

Slide 126

Slide 126 text

Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live Deep Behavioral UX Insight

Slide 127

Slide 127 text

Opinions Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 128

Slide 128 text

Ad Agencies Love It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 129

Slide 129 text

Jared Spool Hates It Conceptual For A Long Time Self-Moderated Moderated Web Analytics Automated Static Automated Live

Slide 130

Slide 130 text

No content

Slide 131

Slide 131 text

Mobile Conceptual For A Long Time Self-Moderated Moderated Automated Static Automated Live }

Slide 132

Slide 132 text

Remote Research Toolkit UNMODERATED (AKA AUTOMATED)

Slide 133

Slide 133 text

3 parts Provide test target Record user behavior Capture comments (optional)

Slide 134

Slide 134 text

USABILLA Clicks & Questions Heatmaps You do your own recruiting $49 - $199

Slide 135

Slide 135 text

No content

Slide 136

Slide 136 text

No content

Slide 137

Slide 137 text

TITLE TEXT LOOP11 (UN-MODERATED)

Slide 138

Slide 138 text

TITLE TEXT

Slide 139

Slide 139 text

GOTOMEETING - $39/MO

Slide 140

Slide 140 text

No content

Slide 141

Slide 141 text

No content

Slide 142

Slide 142 text

TITLE TEXT

Slide 143

Slide 143 text

No content

Slide 144

Slide 144 text

No content

Slide 145

Slide 145 text

No content

Slide 146

Slide 146 text

No content

Slide 147

Slide 147 text

No content

Slide 148

Slide 148 text

No content

Slide 149

Slide 149 text

WHAT WE USE. A LOT. • Ethnio + GTM • Usabilla + Usertesting.com • Guerrilla mobile

Slide 150

Slide 150 text

EXERCISE #3 usertesting.com - 1 free user CODE: uxlisbon2012 usabilla - 2 months free (Sign up for an account, but cancel on the payment page. Send an email to support@usabilla.com and mention UXLX) ethnio - 1 month free or trial. CODE: WORKSHOPLOVE Either “unmoderated” or “practice interviewing”

Slide 151

Slide 151 text

No content

Slide 152

Slide 152 text

RESULTS

Slide 153

Slide 153 text

REMEMBER, BE graceful When All Your TECHNOLOGY BREAKS

Slide 154

Slide 154 text

WHAT THEY WORRY ABOUT

Slide 155

Slide 155 text

WHO ARE YOU AND CAN I TRUST YOU?

Slide 156

Slide 156 text

IF YOU CAN SEE MY SCREEN, CAN YOU SEE MY FACE?

Slide 157

Slide 157 text

WILL YOU BE ABLE TO TAKE CONTROL OF MY COMPUTER?

Slide 158

Slide 158 text

WILL YOU HELP ME GET THE PLUG-IN OFF MY COMPUTER AT THE END?

Slide 159

Slide 159 text

WILL THIS WORK IF I HAVE A FIREWALL/PROXY/STRICT IT DEPARTMENT?

Slide 160

Slide 160 text

IS THIS GOING TO END UP ON YOUTUBE?

Slide 161

Slide 161 text

OXYMORON? REMOTE EMPATHY

Slide 162

Slide 162 text

Taking Notes Second monitor if possible Excel Hack (remoteresear.ch/samples)

Slide 163

Slide 163 text

Observer Involvement IM Campﬁre (!) Integrated Chat Passing Notes

Slide 164

Slide 164 text

HYBRID STUDIES Numbers actually do lie sometimes And they don’t tell you the why 3-5 remote interviews is all you need

Slide 165

Slide 165 text

So, what could go wrong?

Slide 166

Slide 166 text

3 points of failure Technology yours participant’s observer’s Recruiting Test Design

Slide 167

Slide 167 text

Control what you can control Which is, your set up. You wouldn’t go backpacking without checking your gear. Do it 24 hours in advance. Or else.

Slide 168

Slide 168 text

Prepare for what you can’t control User’s dog trips over power cord Internet aﬀected by local weather User can’t install Doesn’t know how to install

Slide 169

Slide 169 text

Mid-Session Tech Problems

Slide 170

Slide 170 text

Live Recruiting Problems What if they gave a study and nobody came?

Slide 171

Slide 171 text

Increase exposure When you’re not getting enough views When the wrong type of users are responding

Slide 172

Slide 172 text

Increase incentives When it’s cheaper When you’re almost getting enough When segments are tough

Slide 173

Slide 173 text

Switch methods When base traﬃc level isn’t high enough

Slide 174

Slide 174 text

Script Problems Cut the BS.

Slide 175

Slide 175 text

ADVANCED TECHNIQUES One to many moderation Multi-threading

Slide 176

Slide 176 text

PORTABLE RESEARCH

Slide 177

Slide 177 text

No content

Slide 178

Slide 178 text

BURN THE SCRIPT FIND OUT WHAT MATTERS.

Slide 179

Slide 179 text

ADVANCED TECHNIQUES High Emotions

Slide 180

Slide 180 text

TITLE TEXT

Slide 181

Slide 181 text

No content

Slide 182

Slide 182 text

No content

Slide 183

Slide 183 text

No content

Slide 184

Slide 184 text

THE FUTURE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE MOBILE

Slide 185

Slide 185 text

DUAL LENS STREAMING FIELD RESEARCH •Livestream •Flip Cam •It’s the view ﬁnder, silly •Actual duct tape

Slide 186

Slide 186 text

No content

Slide 187

Slide 187 text

DISCOUNTS!! Usertesting.com 1 free user. uxlisbon2012 Usabilla 2 months free of any plan. Sign up for an account, but cancel on the payment page. Send an email to support@usabilla.com and mention UXLX Ethnio 1 month free any plan. WORKSHOPLOVE

Slide 188

Slide 188 text

remoteresear.ch/lisbon (look in the slides) DISCOUNTS!! (for, well, that last slide)

Slide 189

Slide 189 text

nate@boltpeters.com @boltron