A/B Testing for Developers

Slide 1

Slide 1 text

Ab Testing

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

What works? motivation aesthetics flow multimedia design graphics speed testimonials simplicity

Slide 4

Slide 4 text

Testing

Slide 5

Slide 5 text

How to do it

Slide 6

Slide 6 text

Metrics

Slide 7

Slide 7 text

Experiments

Slide 8

Slide 8 text

Iterations hypothesis run test decide PROFIT!!

Slide 9

Slide 9 text

• JavaScript-based • Integrates w/Analytics • Supports multivariate testing https://www.google.com/analytics/siteopt/

Slide 10

Slide 10 text

• Rails plugin • Defunct? http://github.com/paulmars/seven_minute_abs

Slide 11

Slide 11 text

• A/B testing for Rails • Uses Redis • Installs as a gem • Integrates with Analytics http://vanity.labnotes.org

Slide 12

Slide 12 text

• A/B testing for Rails • Installs as a plugin • Uses ActiveRecord http://bingocardcreator.com/abingo

Slide 13

Slide 13 text

• Radiant extension • Integrates w/Vanity • Uses ActiveRecord http://ext.radiantcms.org/extensions/227-metrics

Slide 14

Slide 14 text

Roll your own?

Slide 15

Slide 15 text

Installation & Usage

Slide 16

Slide 16 text

What to test? user count surveys logos button size screencasts button color security badge popups wording form size

Slide 17

Slide 17 text

Permutations • A/B/N • A/B/A • Multivariate • Taguchi methods • Heatmaps

Slide 18

Slide 18 text

Pros • It’s easy • Fast setup • Small tweaks • Bang for buck • Reliability

Slide 19

Slide 19 text

Cons • Time consuming (see next slide) • Over reliance • Dishonest? http://www.codinghorror.com/blog/2010/07/ groundhog-day-or-the-problem-with-ab-testing.html

Slide 20

Slide 20 text

Minimizing Time • A/B over A/B/N or A/B/A • Have lots of traffic

Slide 21

Slide 21 text

A/B testing is like sandpaper. You can use it to smooth out details, but you can't actually create anything with it. Nathan Bowers

Slide 22

Slide 22 text

Use your brain

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

Anecdotes • Client signup button • Useless experiments • Login/session error http://github.com/assaf/vanity/issues#issue/10 http://vanity.labnotes.org/identity.html

Slide 30

Slide 30 text

Go do it.

Slide 31

Slide 31 text

Attributions http://flickr.com aussiegall, d_vdm, euthman, missbossy, ka_tate http://alegion-stock.deviantart.com http://ca.askmen.com Matthew Bass twitter @pelargir http://matthewbass.com