What do you mean you don't test in production - we experiment daily

356c244bf4c7a1269e6f2e02705438eb?s=47 Elabor8
September 22, 2015

What do you mean you don't test in production - we experiment daily

by Andrew Thorpe

Step through real life examples of the build, measure, learn process that Open Universities Australia put in place after their MVP launch of a new business Open Training Institute. Learn the actual A/B testing process that was used at Open Training Institute - Selecting a test, building & executing, measuring and productionising a successful test.

Using the actual A/B tests conducted at Open Training Institute we will demonstrate how each stage of the process works, which you can then use to begin your own experimentation journey.

356c244bf4c7a1269e6f2e02705438eb?s=128

Elabor8

September 22, 2015
Tweet

Transcript

  1. 1 awthorpe   stevemac  

  2. 2 awthorpe   stevemac   How to confidently test in

    Prod
  3. 3 awthorpe   stevemac   How it all started

  4. 4 awthorpe   stevemac   Our evolution of Testing in

    Production 1. Learn 2. Understanding 3. Self Confidence 4. Wider Audience 5. Responsibility
  5. 5 awthorpe   stevemac   1. Learn •  State: – 

    Attempting to introduce A/B –  No experience –  No tool –  No team –  No confidence
  6. 6 awthorpe   stevemac   Phase 1 - Test • 

    Start/enroll button
  7. 7 awthorpe   stevemac   Learn

  8. 8 awthorpe   stevemac   Key Learning •  Learn how

    to test, don’t worry about successful tests •  Use an existing tool •  Apply rigor of science, set success criteria before •  Start very small – no impact to business •  Successful test – what now? –  Optimizely 100%
  9. 9 awthorpe   stevemac   2. Understanding •  State: chaotic

    –  Had a tool –  Could measure basic metrics –  Some idea of what success is •  Adhoc selection of experiments •  Adhoc running of tests •  Lack of rigor around the testing process •  No regularity of the tests
  10. 10 awthorpe   stevemac  

  11. 11 awthorpe   stevemac   Understanding Failure

  12. 12 awthorpe   stevemac   Key Learning •  Focus the

    test to a specific issue/goal •  Create cadence -> Weekly meeting (Process) •  Defined length of all tests -> Focus •  Develop a backlog -> process of refining & prioritising •  Successful test, still what to do? –  BA involvement earlier
  13. 13 awthorpe   stevemac   3. Self Confidence •  State:

    –  Knew how to run the testing machine –  Basic governance process –  Running out team capacity –  Tests getting more complex, •  Bigger effort to create •  Larger business impact
  14. 14 awthorpe   stevemac   Self Confidence •  Simple test

    –  Breadcrumbs
  15. 15 awthorpe   stevemac   Key learning •  Complex tests

    require dedicated resources •  Optimization of process •  Focus not just on sales uplift •  Successful test, how do we get to Prod. faster?? - Priority implementation lane
  16. 16 awthorpe   stevemac   Need repeated success (or you’re

    going no where)
  17. 17 awthorpe   stevemac   4. Wider audience •  State:

    –  Limited set of people involved & aware (insular) –  Achieving worthwhile success –  Didn’t have authority to change riskier elements (high business impact) –  Gaining wider business visibility, but not the whole picture –  Negative external view, cowboys testing in prod
  18. 18 awthorpe   stevemac   Wider audience

  19. 19 awthorpe   stevemac   Key learning •  Sell success

    outside the immediate team –  Gain business and exec support •  Sell failure as much as success –  Learning more from failure •  High visibility & transparency –  Wiki, A3 •  Cheaper to test than full dev build
  20. 20 awthorpe   stevemac   5. Responsibility •  State: – 

    We are are here today –  Understand our tools •  Documentation, prioritisation, what tests –  Defined practice •  Confidence in ability to deliver value •  We have external stakeholder buy in
  21. 21 awthorpe   stevemac   Phase 5 – Test • 

    Test example –  Home redesign •  External agency design options –  Check out redesign –  Smoke and mirrors promo discount
  22. 22 awthorpe   stevemac   Phase 5 – Test • 

    Test example –  Home redesign •  External agency design options –  Check out redesign –  Smoke and mirrors promo discount
  23. 23 awthorpe   stevemac   Key learning •  Multiple experiments

    •  Authority to impact high value assets •  Accountability to spend $$$ on external design work
  24. 24 awthorpe   stevemac   What it looks like now

    – 
  25. 25 awthorpe   stevemac   Team Structure •  Cross functional

    team - Business Stakeholder - Business Analyst - Technical Implementation •  Experiment developer, Tester, UI/UX - Marketing - Analytics - Product Owner •  Dedicated team
  26. 26 awthorpe   stevemac   Meeting •  Structure: - Review and

    accept/reject current, - Prioritize proposed tests - Review test implementation •  Time: 30min – 1hr , set time and location •  Ensure meeting is sign off only •  Responsibilities: - Rules: can say yes, BUT have to be there - No external blockers
  27. 27 awthorpe   stevemac   Weekly activities •  Review and

    design of new –  BA input into proposed test –  Implement next tests on dev environment •  Monitoring of current tests –  Kill switch – regular review –  Early identification of success (heads up for BA) •  Implementation of successful (productionise) –  Optimizely 100% –  Fast lane to dev, but not everything
  28. think sharp The image cannot be displayed. Your computer may

    not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again. Thank  you   awthorpe   stevemac  
  29. think sharp Innovation, Agile Transformation, Project Delivery? Looking for a

    career in We’re hiring! Visit: elabor8.com.au/careers
  30. 30 awthorpe   stevemac   Image References •  Nowhere [image]

    Available at: http://4.bp.blogspot.com/-dntZUkFcHPw/UgPVOQ6rCxI/ AAAAAAAAAtQ/Y_IkR6dyjfA/s1600/8220nowhere8221-by-endre-to%CC%81t-1980.png [Accessed 31st August] •  20th Century Fox, (2015) Where are we today – Futurama Fry [image] Available at: http:// memegenerator.net/instance2/1565256 [Accessed 31st August]