A/B Testing to Create Happy Users

Slide 1

Slide 1 text

A/B Testing to Create Happy Users dyn.com/abtest for slides, links and extra stu f

Slide 2

Slide 2 text

@laraswanson

Slide 3

Slide 3 text

@laraswanson We have no idea what's going on in our users' heads.

Slide 4

Slide 4 text

@laraswanson We rarely check to see if guesses about our users are right.

Slide 5

Slide 5 text

@laraswanson We have some options. eye-tracking ﬁeld studies surveys/interviews A/B testing

Slide 6

Slide 6 text

@laraswanson We have some options. eye-tracking ﬁeld studies surveys/interviews A/B testing

Slide 7

Slide 7 text

@laraswanson illustration by Kevin Cornell

Slide 8

Slide 8 text

@laraswanson illustration by Kevin Cornell Content

Slide 9

Slide 9 text

@laraswanson illustration by Kevin Cornell Content Design

Slide 10

Slide 10 text

@laraswanson illustration by Kevin Cornell Content Design Layout

Slide 11

Slide 11 text

@laraswanson illustration by Kevin Cornell Content Design Layout Action

Slide 12

Slide 12 text

A/B testing may be what you need.

Slide 13

Slide 13 text

@laraswanson no user survey could give you this info

Slide 14

Slide 14 text

@laraswanson subtle, continuous improvements

Slide 15

Slide 15 text

@laraswanson solve a design disagreement

Slide 16

Slide 16 text

@laraswanson unique to your site and users

Slide 17

Slide 17 text

Getting Started

Slide 18

Slide 18 text

@laraswanson Test screams (not whispers)

Slide 19

Slide 19 text

@laraswanson Test screams (not whispers) call to action text form length subject lines 41 shades of blue landing page design

Slide 20

Slide 20 text

@laraswanson Test screams (not whispers) call to action text form length subject lines 41 shades of blue landing page design

Slide 21

Slide 21 text

@laraswanson Deﬁne your audience

Slide 22

Slide 22 text

@laraswanson Deﬁne your audience undergrad/grad site goals geography desktop/mobile academic program

Slide 23

Slide 23 text

@laraswanson Deﬁne success

Slide 24

Slide 24 text

@laraswanson Deﬁne success form conversions pages per visit churn rate donations error rate

Slide 25

Slide 25 text

Guess the winner!

Slide 26

Slide 26 text

@laraswanson http://whichtestwon.com/archives/7526 A B

Slide 27

Slide 27 text

@laraswanson http://whichtestwon.com/archives/7526 19% more conversions B

Slide 28

Slide 28 text

@laraswanson Give it enough time me

Slide 29

Slide 29 text

@laraswanson Give it enough time Kill a dead test

Slide 30

Slide 30 text

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A B CALCULATE

Slide 31

Slide 31 text

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 B 180 CALCULATE

Slide 32

Slide 32 text

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180 52 CALCULATE

Slide 33

Slide 33 text

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180 52 99.9% certain B is the winner!

Slide 34

Slide 34 text

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180 24 CALCULATE

Slide 35

Slide 35 text

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180 24 no clear winner :(

Slide 36

Slide 36 text

Technical Stu f

Slide 37

Slide 37 text

@laraswanson http://www.whichmvt.com/ A/B testing technology

Slide 38

Slide 38 text

@laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ﬁculty

Slide 39

Slide 39 text

@laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ﬁculty ease of editing by non-techies

Slide 40

Slide 40 text

@laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ﬁculty ease of editing by non-techies page load time

Slide 41

Slide 41 text

@laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ﬁculty ease of editing by non-techies page load time integration with analytics

Slide 42

Slide 42 text

@laraswanson Audience segmentation academic program mobile/desktop geography

Slide 43

Slide 43 text

@laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer

Slide 44

Slide 44 text

@laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests

Slide 45

Slide 45 text

@laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA

Slide 46

Slide 46 text

@laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA calculates signiﬁcance

Slide 47

Slide 47 text

@laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA calculates signiﬁcance page load time-optimized

Slide 48

Slide 48 text

@laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA calculates signiﬁcance page load time-optimized can do multiple tests on one page

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Guess the winner!

Slide 51

Slide 51 text

@laraswanson http://whichtestwon.com/archives/2450 A B

Slide 52

Slide 52 text

@laraswanson http://whichtestwon.com/archives/2450 73% more conversions A

Slide 53

Slide 53 text

Google Analytics integration

Slide 54

Slide 54 text

@laraswanson Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]);

Slide 55

Slide 55 text

Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson custom variable slot custom variable slot

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

@laraswanson Google Analytics > Audience > Custom > Custom Variables

Slide 60

Slide 60 text

@laraswanson

Slide 61

Slide 61 text

@laraswanson

Slide 62

Slide 62 text

@laraswanson

Slide 63

Slide 63 text

Guess the winner!

Slide 64

Slide 64 text

@laraswanson http://whichtestwon.com/archives/8413 A B

Slide 65

Slide 65 text

@laraswanson http://whichtestwon.com/archives/8413 More PhD candidates submitted request for information B

Slide 66

Slide 66 text

A/B tests will make you mad.

Slide 67

Slide 67 text

@laraswanson valuable development time

Slide 68

Slide 68 text

@laraswanson may not reach signiﬁcant results

Slide 69

Slide 69 text

@laraswanson you'll be excited and then maybe disappointed

Slide 70

Slide 70 text

@laraswanson A/B tests will prove you wrong

Slide 71

Slide 71 text

A/B tests will make your users happy.

Slide 72

Slide 72 text

@laraswanson your site will be easier to use

Slide 73

Slide 73 text

@laraswanson users will accomplish what they came to do

Slide 74

Slide 74 text

A/B tests will make your stakeholders happy.

Slide 75

Slide 75 text

@laraswanson better conversion rates

Slide 76

Slide 76 text

@laraswanson alumni engagement

Slide 77

Slide 77 text

@laraswanson insight for each audience segment

Slide 78

Slide 78 text

A/B tests will make you mad happy.

Slide 79

Slide 79 text

@laraswanson knowing why is better than knowing how

Slide 80

Slide 80 text

@laraswanson impress your stakeholders

Slide 81

Slide 81 text

@laraswanson data to say "NO" to bad ideas

Slide 82

Slide 82 text

Talking to stakeholders

Slide 83

Slide 83 text

@laraswanson Stakeholder: I heard from a large donor that it's hard to donate online.

Slide 84

Slide 84 text

@laraswanson Decoder ring: Online donations are down, someone complained about the online experience, I'm lost when it comes to the web so I need your help.

Slide 85

Slide 85 text

@laraswanson You: I'm ready to help! We can actually test a few di ferent versions of the form to see which gets more donations. Decoder ring: Online donations are down, someone complained about the online experience, I'm lost when it comes to the web so I need your help.

Slide 86

Slide 86 text

@laraswanson Stakeholder: Really? You can't just ﬁx it right now?

Slide 87

Slide 87 text

@laraswanson Decoder ring: I don't understand why we'd wait any longer.

Slide 88

Slide 88 text

@laraswanson You: By running a test we can statistically determine which version of the form is better. Decoder ring: I don't understand why we'd wait any longer.

Slide 89

Slide 89 text

@laraswanson You: It avoids putting up a ﬁx now that may actually make things worse, because we won't have any data about how users respond to it. Decoder ring: I don't understand why we'd wait any longer.

Slide 90

Slide 90 text

@laraswanson Stakeholder: I don't understand. How would people see the di ferent versions?

Slide 91

Slide 91 text

@laraswanson Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!

Slide 92

Slide 92 text

@laraswanson You: Half of visitors will see a new version that I think will make our users happier and more likely to donate. Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!

Slide 93

Slide 93 text

@laraswanson You: The other half will see the current form. I'll measure donations from each version so we can see which performs better. Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!

Slide 94

Slide 94 text

@laraswanson Stakeholder: And then what? Then you'll ﬁx it?

Slide 95

Slide 95 text

@laraswanson Decoder ring: I just want to hear that you're going to ﬁx this.

Slide 96

Slide 96 text

@laraswanson You: You bet. And I can continue to run tests on it to improve it more and more over time. Decoder ring: I just want to hear that you're going to ﬁx this.

Slide 97

Slide 97 text

@laraswanson You: We can make the form better than it's ever been and because we'll have data, we'll be conﬁdent in its results. Decoder ring: I just want to hear that you're going to ﬁx this.

Slide 98

Slide 98 text

Guess the winner!

Slide 99

Slide 99 text

@laraswanson http://whichtestwon.com/archives/14918 A B

Slide 100

Slide 100 text

@laraswanson http://whichtestwon.com/archives/14918 57% more leads B

Slide 101

Slide 101 text

@laraswanson http://whichtestwon.com/archives/4781 A B

Slide 102

Slide 102 text

@laraswanson http://whichtestwon.com/archives/4781 A B

Slide 103

Slide 103 text

@laraswanson http://whichtestwon.com/archives/4781 A B

Slide 104

Slide 104 text

@laraswanson http://whichtestwon.com/archives/4781 17% more conversions Wednesday and Thursday a ternoons B

Slide 105

Slide 105 text

@laraswanson http://whichtestwon.com/archives/4781 22% more conversions on weekday and weekend mornings A

Slide 106

Slide 106 text

@laraswanson http://whichtestwon.com/archives/2655 A B

Slide 107

Slide 107 text

@laraswanson http://whichtestwon.com/archives/2655 86% more "Subscribe" clicks A

Slide 108

Slide 108 text

@laraswanson http://whichtestwon.com/archives/7453 A B

Slide 109

Slide 109 text

@laraswanson http://whichtestwon.com/archives/7453 2666% higher CTR B

Slide 110

Slide 110 text

@laraswanson http://whichtestwon.com/archives/3026 A B

Slide 111

Slide 111 text

@laraswanson http://whichtestwon.com/archives/3026 25% more form submissions A

Slide 112

Slide 112 text

How did you do?

Slide 113

Slide 113 text

@laraswanson https://twitter.com/katiedel/status/240136499704778754 "Only 10-30% (of 1000's) of experiments improved the metrics they were designed the improve"

Slide 114

Slide 114 text

That's why we need to test.

Slide 115

Slide 115 text

@laraswanson Alternatives to A/B testing

Slide 116

Slide 116 text

@laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt

Slide 117

Slide 117 text

@laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt Multi-armed bandit: http://bit.ly/multi-armed-bandit

Slide 118

Slide 118 text

@laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt Multi-armed bandit: http://bit.ly/multi-armed-bandit Tons of tools: http://whichtestwon.com/techguide

Slide 119

Slide 119 text

Thanks! Questions? dyn.com/abtest @laraswanson dyn.com/careers - we're hiring UI devs!