Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What do you mean you don't test in production - we experiment daily

Elabor8
September 22, 2015

What do you mean you don't test in production - we experiment daily

by Andrew Thorpe

Step through real life examples of the build, measure, learn process that Open Universities Australia put in place after their MVP launch of a new business Open Training Institute. Learn the actual A/B testing process that was used at Open Training Institute - Selecting a test, building & executing, measuring and productionising a successful test.

Using the actual A/B tests conducted at Open Training Institute we will demonstrate how each stage of the process works, which you can then use to begin your own experimentation journey.

Elabor8

September 22, 2015
Tweet

More Decks by Elabor8

Other Decks in Technology

Transcript

  1. 4 awthorpe   stevemac   Our evolution of Testing in

    Production 1. Learn 2. Understanding 3. Self Confidence 4. Wider Audience 5. Responsibility
  2. 5 awthorpe   stevemac   1. Learn •  State: – 

    Attempting to introduce A/B –  No experience –  No tool –  No team –  No confidence
  3. 8 awthorpe   stevemac   Key Learning •  Learn how

    to test, don’t worry about successful tests •  Use an existing tool •  Apply rigor of science, set success criteria before •  Start very small – no impact to business •  Successful test – what now? –  Optimizely 100%
  4. 9 awthorpe   stevemac   2. Understanding •  State: chaotic

    –  Had a tool –  Could measure basic metrics –  Some idea of what success is •  Adhoc selection of experiments •  Adhoc running of tests •  Lack of rigor around the testing process •  No regularity of the tests
  5. 12 awthorpe   stevemac   Key Learning •  Focus the

    test to a specific issue/goal •  Create cadence -> Weekly meeting (Process) •  Defined length of all tests -> Focus •  Develop a backlog -> process of refining & prioritising •  Successful test, still what to do? –  BA involvement earlier
  6. 13 awthorpe   stevemac   3. Self Confidence •  State:

    –  Knew how to run the testing machine –  Basic governance process –  Running out team capacity –  Tests getting more complex, •  Bigger effort to create •  Larger business impact
  7. 15 awthorpe   stevemac   Key learning •  Complex tests

    require dedicated resources •  Optimization of process •  Focus not just on sales uplift •  Successful test, how do we get to Prod. faster?? - Priority implementation lane
  8. 17 awthorpe   stevemac   4. Wider audience •  State:

    –  Limited set of people involved & aware (insular) –  Achieving worthwhile success –  Didn’t have authority to change riskier elements (high business impact) –  Gaining wider business visibility, but not the whole picture –  Negative external view, cowboys testing in prod
  9. 19 awthorpe   stevemac   Key learning •  Sell success

    outside the immediate team –  Gain business and exec support •  Sell failure as much as success –  Learning more from failure •  High visibility & transparency –  Wiki, A3 •  Cheaper to test than full dev build
  10. 20 awthorpe   stevemac   5. Responsibility •  State: – 

    We are are here today –  Understand our tools •  Documentation, prioritisation, what tests –  Defined practice •  Confidence in ability to deliver value •  We have external stakeholder buy in
  11. 21 awthorpe   stevemac   Phase 5 – Test • 

    Test example –  Home redesign •  External agency design options –  Check out redesign –  Smoke and mirrors promo discount
  12. 22 awthorpe   stevemac   Phase 5 – Test • 

    Test example –  Home redesign •  External agency design options –  Check out redesign –  Smoke and mirrors promo discount
  13. 23 awthorpe   stevemac   Key learning •  Multiple experiments

    •  Authority to impact high value assets •  Accountability to spend $$$ on external design work
  14. 25 awthorpe   stevemac   Team Structure •  Cross functional

    team - Business Stakeholder - Business Analyst - Technical Implementation •  Experiment developer, Tester, UI/UX - Marketing - Analytics - Product Owner •  Dedicated team
  15. 26 awthorpe   stevemac   Meeting •  Structure: - Review and

    accept/reject current, - Prioritize proposed tests - Review test implementation •  Time: 30min – 1hr , set time and location •  Ensure meeting is sign off only •  Responsibilities: - Rules: can say yes, BUT have to be there - No external blockers
  16. 27 awthorpe   stevemac   Weekly activities •  Review and

    design of new –  BA input into proposed test –  Implement next tests on dev environment •  Monitoring of current tests –  Kill switch – regular review –  Early identification of success (heads up for BA) •  Implementation of successful (productionise) –  Optimizely 100% –  Fast lane to dev, but not everything
  17. think sharp The image cannot be displayed. Your computer may

    not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again. Thank  you   awthorpe   stevemac  
  18. think sharp Innovation, Agile Transformation, Project Delivery? Looking for a

    career in We’re hiring! Visit: elabor8.com.au/careers
  19. 30 awthorpe   stevemac   Image References •  Nowhere [image]

    Available at: http://4.bp.blogspot.com/-dntZUkFcHPw/UgPVOQ6rCxI/ AAAAAAAAAtQ/Y_IkR6dyjfA/s1600/8220nowhere8221-by-endre-to%CC%81t-1980.png [Accessed 31st August] •  20th Century Fox, (2015) Where are we today – Futurama Fry [image] Available at: http:// memegenerator.net/instance2/1565256 [Accessed 31st August]