Unit 4 - An Experimental Approach to Product Development

D8004857fc10614cfa4dec1bae20f874?s=47 Jez Humble
September 24, 2018

Unit 4 - An Experimental Approach to Product Development

This class will present hypothesis-driven development, the cutting-edge paradigm for evolving validated products. We’ll dive into how to frame hypotheses, design experiments, and use A/B testing to gather data to prove or disprove our ideas.

D8004857fc10614cfa4dec1bae20f874?s=128

Jez Humble

September 24, 2018
Tweet

Transcript

  1. i290 lean/agile product management unit 4: experimental product development @jezhumble

    https://leanagile.pm/ humble@berkeley.edu This work © 2015-2018 Jez Humble Licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
  2. identify experiments to test hypotheses understand how to do outcome-based

    planning describe hypothesis-driven development understand why small batches are important define A/B testing and the culture it enables learning outcomes
  3. Epic Theme Story

  4. minimize output, maximize outcome Jeff Patton, User Story Mapping p.

    xlii
  5. impact mapping Gojko Adzic, Impact Mapping

  6. working backwards http://www.allthingsdistributed.com/2006/11/working_backwards.html

  7. @jezhumble Jeff Gothelf “Better product definition with Lean UX and

    Design” http://bit.ly/TylT6A hypothesis-driven delivery We believe that [building this feature] [for these people] will achieve [this outcome]. We will know we are successful when we see [this signal from the market].
  8. None
  9. COST OF EXPERIMENTS 9 Production Software SPEED COST new services

    feasibility spike service substitution integration Quantitative forecasting real-time price experiment Data sampling and modeling tests Sketches & Paper Prototypes Interactive Prototype Software demo Interviews & surveys micro-niche Wizard of Oz VIABILITY (BUSINESS) | DESIRABILITY (CUSTOMER) | FEASIBILITY (TECH)
  10. exercise • choose a hypothesis from week 2’s class •

    design an experiment to test your hypothesis • what do you expect the results to be? • what result will confirm your hypothesis? • what result will disprove your hypothesis? • how soon can we get the result?
  11. “Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn

    Thomas | http://bit.ly/19Z5izI
  12. “Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn

    Thomas | http://bit.ly/19Z5izI
  13. “Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn

    Thomas | http://bit.ly/19Z5izI
  14. Jon Jenkins, “Velocity Culture, The Unmet Challenge in Ops” 2011

    | http://bit.ly/1vJo1Ya
  15. do less “Evaluating well-designed and executed experiments that were designed

    to improve a key metric, only about 1/3 were successful at improving the key metric!” “Online Experimentation at Microsoft” | Kohavi et al | http://stanford.io/130uW6X
  16. “I think building this culture is the key to innovation.

    Creativity must flow from everywhere. Whether you are a summer intern or the CTO, any good idea must be able to seek an objective test, preferably a test that exposes the idea to real customers. Everyone must be able to experiment, learn, and iterate.” http://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html
  17. @jezhumble WORK IN SMALL BATCHES WORK IN SMALL BATCHES WORK

    IN SMALL BATCHES WORK IN SMALL BATCHES WORK IN SMALL BATCHES WORK IN SMALL BATCHES WORK IN SMALL BATCHES WORK IN SMALL BATCHES WORK IN SMALL BATCHES
  18. less rework, higher quality you can stop at any time

    with a working system faster feedback (assuming people pay attention) higher motivation quickly release high priority features / bugfixes working in small batches Don Reinertsen, Principles of Product Development Flow, ch5.
  19. further reading https://www.infoq.com/presentations/controlled-experiments https://svpg.com/assets/Files/goodprd.pdf Tom DeMarco & Tim Lister, Waltzing

    with Bears Humble et al, Lean Enterprise ch 9 Gojko Adzic, Impact Mapping