Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Finance@Google2015 Simply Business - Living the Test Culture

Finance@Google2015 Simply Business - Living the Test Culture

A talk I gave on how and when we test at Simply Business, and some of the challenges.

8b2d2d488fa1d4ca40a2b3e4f8991799?s=128

Lukas Oberhuber

September 16, 2015
Tweet

Transcript

  1. Case: Living the Test Culture Lukas Oberhuber | CTO -

    Simply Business
  2. 55,000,000 € Revenues 300 Employees 300,000 Policies

  3. None
  4. flickr photo by Pink Sherbet Photography http://flickr.com/photos/pinksherbet/3370498053 shared under a

    Creative Commons (BY) license
  5. The Principles of Living a Test Culture Build, Measure, Learn

    Test What’s Important How You Should Roll Be Bold It’s Hard
  6. Foundation to Test Build Learn Measure

  7. What we test Everything Everything Everything Everything Everything Everything Everything

  8. How we set up Tests • Decide what to learn

    (Hypothesis) • Write experiment (Measures) • Build and run experiment • Results • Pivot or persevere
  9. How to Ensure Accurate Data • Not easy • Biases

    and missing skills get in the way • Avoid post hoc analysis
  10. Results “You will walk through the valley of the shadow

    of death” 26% YoY growth in conversion through incremental change
  11. Comparison Pages Multi-month build Driven by customer requests Good usability

    results A/B Test showed 3 point uplift
  12. How should you Roll? Sell policies via brokers • Small

    investment • Failed • New hypothesis • None worked • Pivoted Test what’s important, not what’s easy
  13. Be Bold Source https://www.123-reg.co.uk/blog/top-6-lists/six-ab-tests-to-skyrocket-your-sales/

  14. Be Bold Small tests, small results People win the lottery

    all the time! Could it be? Yes it could. Something’s coming, something good. Gonna be great. Source https://www.123-reg.co.uk/blog/top-6-lists/six-ab-tests-to-skyrocket-your-sales/
  15. How should you Roll? Look beyond the Comfortable And A/B

    testing is only one of the methods Source: http://www.nngroup.com/articles/which-ux-research-methods/
  16. flickr photo by stephen w morris http://flickr.com/photos/nonlin/4013035510 shared under a

    Creative Commons (BY) license Culture
  17. A Word about HiPPOs

  18. THANKS! Lukas Oberhuber lukas.oberhuber@simplybusiness.co.uk www.simplybusiness.co.uk @lukasco