AB Testing - Working out your apps to make more $$$

Slide 1

Slide 1 text

working out your apps to make more $$$ Ab Testing

Slide 2

Slide 2 text

GET /ME • operations @ seatgeek.com • cakephp core developer • blog @ josediazgonzalez.com • github @ josegonzalez • irc @ savant • tweet @ savant

Slide 3

Slide 3 text

How do we make the most money possible? How do we grow as quickly as possible? goals for a site

Slide 4

Slide 4 text

Both ways involve changing how you do things

Slide 5

Slide 5 text

• redesigns • new features • removing features • simplifying user flows things we change

Slide 6

Slide 6 text

how do you know your changes are making an impact?

Slide 7

Slide 7 text

test all the things

Slide 8

Slide 8 text

definitions • Element: A discrete item on a page (button, link, text) • Page: a web page or app view containing your test • Variation: a version of the page that has some changes • Conversion: when a user performs a desired page action

Slide 9

Slide 9 text

what is an ab test?

Slide 10

Slide 10 text

source: wikipedia A/B testing is jargon for a randomized experiment with two variants, A and B, which are the control and treatment in the controlled experiment

Slide 11

Slide 11 text

ab testing is a way to randomly test whether one version of a thing is better than another version

Slide 12

Slide 12 text

ab testing is comparing a control version to an alternate version and seeing what differences there are in conversion

Slide 13

Slide 13 text

ab testing allows you to compare two different sets of visitors and measure conversion changes from each set of users.

Slide 14

Slide 14 text

ab testing is a way to test in production

Slide 15

Slide 15 text

• To ensure your changes are effecting success • Data driven development • To ensure your efforts are minimally wasted why ab test?

Slide 16

Slide 16 text

• Be targeted at a percentage of your traffic • Require less traffic to see effects • Be used in conjunction with heat maps, click tracking etc. ab tests can

Slide 17

Slide 17 text

Things we test • button sizes • colors • page layout • page text • user interaction (some of the)

Slide 18

Slide 18 text

• Adding video to your landing page • Raising conversion points above the fold • Nouns, ie. Masculine vs Feminine vs Unspecified • Positive vs Negative • Form Length, required vs non-required fields Things you can test

Slide 19

Slide 19 text

• You need a representative sample size • Must control for daily/seasonal traffic differences • Must control for skews in available data how long should i run tests for?

Slide 20

Slide 20 text

1. Brainstorm potential testing targets 2. Create test variants 3. Launch tests and drive traffic to your pages 4. Promote a test winner (and analyze results) 5. Start a new test testing workflow

Slide 21

Slide 21 text

jUDGING TEST RESULTS I understand the results I do not understand the results Conversion Improvement Woo Hoo! You are a genius. Start your next test. ! (Pass) Really? Ok, do a test to figure out the next step. ! (Pass, kinda) Conversion Decline Good save. Stop the test and try a new hypothesis. ! (Pass) Seriously, a kitten just died because of you. ! (Fail) source: http://files.meetup.com/5254712/ultimate-AB-testing-guide.pdf

Slide 22

Slide 22 text

• each variant has a certain (unknown) conversion rate • alternatives can be “chosen” via exploitation or exploration routes • given more traffic to a variant when it has a higher conversion rate Multi-armed bandit

Slide 23

Slide 23 text

• each variant more than one element changed across • requires more traffic to reach a confidence level • harder to integrate with 3rd-party tracking tools • failures are noticeable multi-variate tests

Slide 24

Slide 24 text

a real life case study Testing Mobile checkouts

Slide 25

Slide 25 text

mobile user funnel As a User, I will: • click an event link • click a ticket link • click out to market • purchase a ticket

Slide 26

Slide 26 text

first test had two variations

Slide 27

Slide 27 text

Test 1 • Add a view from seat • Larger buy button • More structured ticket info • Centered map at bottom

Slide 28

Slide 28 text

and the old shitty version soundly trounced the new version

Slide 29

Slide 29 text

and it continued to do so for 4 tests

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

the old version was something i designed drunk

Slide 32

Slide 32 text

i am not a smart man

Slide 33

Slide 33 text

test 5 was a slaughter

Slide 34

Slide 34 text

test 6 was similarly embarrassing

Slide 35

Slide 35 text

we gave up at round 7

Slide 36

Slide 36 text

This continues to be completely confounding. Maybe it has something to do with the event info being at the top? - Jack G, SeatGeek Co-Founder

Slide 37

Slide 37 text

ab tests do not need to confirm your suspicions

Slide 38

Slide 38 text

ab tests can just as easily invalidate your hypothesis

Slide 39

Slide 39 text

how do i start?

Slide 40

Slide 40 text

ab testing on a variety of platforms ! github.com/seatgeek/sixpack sixpack

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

• useful for feature flags • json api • multiple kpis features

Slide 43

Slide 43 text

use SeatGeek\Sixpack\Session\Base as Sixpack; ! $sp = new Sixpack; $test = $sp->participate('test', [‘Buy', ‘Download’]); $alternative = $test->getAlternative(); ! $this->Html->link($alternative, ‘/some/link’); start a test in PHP

Slide 44

Slide 44 text

var session = new sixpack.Session(); session.convert("test", function (err, res) { if (err) throw err; $(“.convert”).html(JSON.stringify(res)); }); Convert in js