Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AB Testing - Working out your apps to make more $$$

AB Testing - Working out your apps to make more $$$

An AB Testing talk given at CakeFest 2014 that covers AB testing as an idea, as well as a real-world case study from SeatGeek.com and introducing an open-source ab testing framework.


Jose Diaz-Gonzalez

August 24, 2014


  1. working out your apps to make more $$$ Ab Testing

  2. GET /ME • operations @ seatgeek.com • cakephp core developer

    • blog @ josediazgonzalez.com • github @ josegonzalez • irc @ savant • tweet @ savant
  3. How do we make the most money possible? How do

    we grow as quickly as possible? goals for a site
  4. Both ways involve changing how you do things

  5. • redesigns • new features • removing features • simplifying

    user flows things we change
  6. how do you know your changes are making an impact?

  7. test all the things

  8. definitions • Element: A discrete item on a page (button,

    link, text) • Page: a web page or app view containing your test • Variation: a version of the page that has some changes • Conversion: when a user performs a desired page action
  9. what is an ab test?

  10. source: wikipedia A/B testing is jargon for a randomized experiment

    with two variants, A and B, which are the control and treatment in the controlled experiment
  11. ab testing is a way to randomly test whether one

    version of a thing is better than another version
  12. ab testing is comparing a control version to an alternate

    version and seeing what differences there are in conversion
  13. ab testing allows you to compare two different sets of

    visitors and measure conversion changes from each set of users.
  14. ab testing is a way to test in production

  15. • To ensure your changes are effecting success • Data

    driven development • To ensure your efforts are minimally wasted why ab test?
  16. • Be targeted at a percentage of your traffic •

    Require less traffic to see effects • Be used in conjunction with heat maps, click tracking etc. ab tests can
  17. Things we test • button sizes • colors • page

    layout • page text • user interaction (some of the)
  18. • Adding video to your landing page • Raising conversion

    points above the fold • Nouns, ie. Masculine vs Feminine vs Unspecified • Positive vs Negative • Form Length, required vs non-required fields Things you can test
  19. • You need a representative sample size • Must control

    for daily/seasonal traffic differences • Must control for skews in available data how long should i run tests for?
  20. 1. Brainstorm potential testing targets 2. Create test variants 3.

    Launch tests and drive traffic to your pages 4. Promote a test winner (and analyze results) 5. Start a new test testing workflow
  21. jUDGING TEST RESULTS I understand the results I do not

    understand the results Conversion Improvement Woo Hoo! You are a genius. Start your next test. ! (Pass) Really? Ok, do a test to figure out the next step. ! (Pass, kinda) Conversion Decline Good save. Stop the test and try a new hypothesis. ! (Pass) Seriously, a kitten just died because of you. ! (Fail) source: http://files.meetup.com/5254712/ultimate-AB-testing-guide.pdf
  22. • each variant has a certain (unknown) conversion rate •

    alternatives can be “chosen” via exploitation or exploration routes • given more traffic to a variant when it has a higher conversion rate Multi-armed bandit
  23. • each variant more than one element changed across •

    requires more traffic to reach a confidence level • harder to integrate with 3rd-party tracking tools • failures are noticeable multi-variate tests
  24. a real life case study Testing Mobile checkouts

  25. mobile user funnel As a User, I will: • click

    an event link • click a ticket link • click out to market • purchase a ticket
  26. first test had two variations

  27. Test 1 • Add a view from seat • Larger

    buy button • More structured ticket info • Centered map at bottom
  28. and the old shitty version soundly trounced the new version

  29. and it continued to do so for 4 tests

  30. None
  31. the old version was something i designed drunk

  32. i am not a smart man

  33. test 5 was a slaughter

  34. test 6 was similarly embarrassing

  35. we gave up at round 7

  36. This continues to be completely confounding. Maybe it has something

    to do with the event info being at the top? - Jack G, SeatGeek Co-Founder
  37. ab tests do not need to confirm your suspicions

  38. ab tests can just as easily invalidate your hypothesis

  39. how do i start?

  40. ab testing on a variety of platforms ! github.com/seatgeek/sixpack sixpack

  41. None
  42. • useful for feature flags • json api • multiple

    kpis features
  43. use SeatGeek\Sixpack\Session\Base as Sixpack; ! $sp = new Sixpack; $test

    = $sp->participate('test', [‘Buy', ‘Download’]); $alternative = $test->getAlternative(); ! $this->Html->link($alternative, ‘/some/link’); start a test in PHP
  44. var session = new sixpack.Session(); session.convert("test", function (err, res) {

    if (err) throw err; $(“.convert”).html(JSON.stringify(res)); }); Convert in js
  45. start testing today!

  46. buy my book https://bit.ly/phpcakebook 20% off: CAKEFEST2014

  47. thanks