Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unit 3 - An Experimental Approach to Product Development

D8004857fc10614cfa4dec1bae20f874?s=47 Jez Humble
September 21, 2020

Unit 3 - An Experimental Approach to Product Development

This class will present hypothesis-driven development, the modern paradigm for evolving validated products. We’ll dive into how to frame hypotheses, design experiments, and use A/B testing to gather data to prove or disprove our ideas.


Jez Humble

September 21, 2020


  1. i290 lean/agile product management unit3: experimental product development @jezhumble https://leanagile.pm/

    humble@berkeley.edu This work © 2015-2020 Jez Humble Licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
  2. identify experiments to test hypotheses understand how to do outcome-based

    planning describe hypothesis-driven development understand why small batches are important define A/B testing and the culture it enables learning outcomes
  3. Epic Theme Story

  4. projects: scope, cost, time product: impact / value (e.g. change

    in behavior) project vs product
  5. experts are what they do “Given a representative task in

    the domain, a badass performs in a superior way, more reliably” —Kathy Sierra, Badass
  6. minimize output, maximize outcome Jeff Patton, User Story Mapping p.

  7. impact mapping Gojko Adzic, Impact Mapping

  8. @jezhumble Jeff Gothelf “Better product definition with Lean UX and

    Design” http://bit.ly/TylT6A hypothesis-driven delivery We believe that [building this feature] [for these people] will achieve [this outcome]. We will know we are successful when we see [this signal from the market].
  9. None
  10. working backwards http://www.allthingsdistributed.com/2006/11/working_backwards.html

  11. COST OF EXPERIMENTS 11 Production Software SPEED COST new services

    feasibility spike service substitution integration Quantitative forecasting real-time price experiment Data sampling and modeling tests Sketches & Paper Prototypes Interactive Prototype Software demo Interviews & surveys micro-niche Wizard of Oz VIABILITY (BUSINESS) | DESIRABILITY (CUSTOMER) | FEASIBILITY (TECH)
  12. exercise • choose a hypothesis from week 2’s class •

    design an experiment to test your hypothesis • what do you expect the results to be? • what result will confirm your hypothesis? • what result will disprove your hypothesis? • how soon can we get the result?
  13. a/b testing 50% of visitors see variation A (control) 50%

    of visitors see variation B (treatment) 20% conversion 60% conversion
  14. “Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn

    Thomas | http://bit.ly/19Z5izI
  15. “Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn

    Thomas | http://bit.ly/19Z5izI
  16. “Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn

    Thomas | http://bit.ly/19Z5izI
  17. Jon Jenkins, “Velocity Culture, The Unmet Challenge in Ops” 2011

    | http://bit.ly/1vJo1Ya
  18. do less “Evaluating well-designed and executed experiments that were designed

    to improve a key metric, only about 1/3 were successful at improving the key metric!” “Online Experimentation at Microsoft” | Kohavi et al | http://stanford.io/130uW6X
  19. “I think building this culture is the key to innovation.

    Creativity must flow from everywhere. Whether you are a summer intern or the CTO, any good idea must be able to seek an objective test, preferably a test that exposes the idea to real customers. Everyone must be able to experiment, learn, and iterate.” http://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html
  20. less rework, higher quality you can stop at any time

    with a working system faster feedback (assuming people pay attention) higher motivation quickly release high priority features / bugfixes working in small batches Don Reinertsen, Principles of Product Development Flow, ch5.
  21. lead time “How long would it take your organization to

    deploy a change that involved just one single line of code? Do you do this on a repeatable, reliable basis?” Mary and Tom Poppendieck, Implementing Lean Software Development, p59.
  22. small batches • Reduce cycle time, variability in flow, risk,

    overhead • Accelerates feedback • Increase efficiency, motivation and urgency Don Reinertsen, Principles of Product Development Flow, ch5.
  23. INVEST • Independent: can be worked on in any order

    • Negotiable: a conversation not a contract • Valuable: delivers benefit to a stakeholder • Estimable: to a good degree of precision • Small: less than a week to build • Testable: clearly defined acceptance criteria http://bit.ly/small-batches-invest
  24. experiment design • Identify risky assumption about your product. •

    For example: “in order to make our business model work, 5% of people who sign up for my service that meet my criteria must pay $100 for the service.” • Write down hypothesis. • Design an experiment to test this with small sample size and acceptable level of precision, including testable success criteria. • Gather data. Analyze. State results. • State if hypothesis was validated or not. • What did you learn? What changes will you make?
  25. further reading https://www.infoq.com/presentations/controlled-experiments https://svpg.com/assets/Files/goodprd.pdf Tom DeMarco & Tim Lister, Waltzing

    with Bears Humble et al, Lean Enterprise ch 9 Gojko Adzic, Impact Mapping