Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Solution Validation and Metrics - UBC PM Club

Solution Validation and Metrics - UBC PM Club

March 2021 presentation for the UBC PM Club on solution validation and metrics. Had a ton of fun and turned this into an interactive workshop as well.

28c50520ae65f87899bc12c94e4a7c09?s=128

Scott Baldwin

March 19, 2021
Tweet

Transcript

  1. © 2021 Productboard, Inc. proprietary & confidential Solution Validation (and

    Metrics) UBC Product Management Club
  2. © 2021 Productboard, Inc. proprietary & confidential i Product Excellence

    Consultant scott@productboard.com Scott Baldwin
  3. © 2021 Productboard, Inc. proprietary & confidential Former customer and

    long-time product leader About Scott 🇨🇦 Based in Vancouver, BC, Canada 😀 20+ years experience as a PM and previously in UX research & IA 👨🏻‍💻 Led product teams in a variety of companies — Thinkific, Finning Digital, Central 1, Inovera Solutions, Blue Zone Entertainment 🤝 Active participant in the product community – passionate about coaching product teams and mentoring those looking to enter PM and grow in their careers 📚 Featured in the O’Reilly book Design Leadership 🥁 Music, ultramarathons 🏃🏻‍♂, and family
  4. © 2021 Productboard, Inc. proprietary & confidential 1. Why we

    need to validate 2. Tenets of good validation practices 3. Methods/techniques and how to choose the right one 4. Workshop 5. Q&A 6. Toolkit Does anyone even pay attention to these agenda slides? Agenda 📒
  5. © 2021 Productboard, Inc. proprietary & confidential 🙋🏽‍♂ Why validate?

    Go to www.slido.com and enter code #90055 to share your thoughts
  6. © 2021 Productboard, Inc. proprietary & confidential ⓘ Start presenting

    to display the poll results on this slide. Why should we do solution validation?
  7. © 2021 Productboard, Inc. proprietary & confidential People = who

    Problem = why Product = how Whoa, he’s already making assumptions? My assumptions 🧐
  8. © 2021 Productboard, Inc. proprietary & confidential Most ideas fail

    because... High failure rate
  9. © 2021 Productboard, Inc. proprietary & confidential Valuable = will

    customers buy this or choose to use it? Usable = can the user figure out how to use this? Feasible = can we build this given the time, skills, and technology we have? Viable = does this work for the various aspects of our business Valuable, usable, feasible, and viable Four risks
  10. © 2021 Productboard, Inc. proprietary & confidential Build Measure Learn

    prototype data ideas
  11. © 2021 Productboard, Inc. proprietary & confidential Guidelines

  12. © 2021 Productboard, Inc. proprietary & confidential Think in experiments,

    not projects
  13. © 2021 Productboard, Inc. proprietary & confidential DE&I

  14. © 2021 Productboard, Inc. proprietary & confidential Fast, easy, and

    effective
  15. © 2021 Productboard, Inc. proprietary & confidential Start together, not

    alone
  16. © 2021 Productboard, Inc. proprietary & confidential Get out of

    the building
  17. © 2021 Productboard, Inc. proprietary & confidential 🤨 Choosing the

    right method
  18. © 2021 Productboard, Inc. proprietary & confidential Which of the

    four risks are you validating for? What type of hypothesis are you testing? How much evidence do you already have for the specific hypothesis? How much time do you have until the next major decision point or until you run out of money? How will you know if you’re successful or not? Before you start Questions to ask
  19. © 2021 Productboard, Inc. proprietary & confidential Reduce uncertainty as

    much as possible before you build anything Cheap and fast early in journey Use multiple touchpoints to increase the strength and likelihood of gathering evidence Pick your approach wisely - strongest evidence given constraints that your team can support Keep in mind
  20. © 2021 Productboard, Inc. proprietary & confidential Methods/techniques 🤠

  21. © 2021 Productboard, Inc. proprietary & confidential Time Strength of

    evidence Clickable prototype Single feature MVP Mash up Concierge Landing page Crowdfunding Split test Presale Survey Wizard of Oz Mock sale Letter of intent Pop up store Spike Usability testing Guerilla testing
  22. © 2021 Productboard, Inc. proprietary & confidential Multi-question survey used

    to learn Surveys
  23. © 2021 Productboard, Inc. proprietary & confidential Digital UI with

    clickable zones to represent the software’s reactions to the customer’s interactions Clickable prototype
  24. © 2021 Productboard, Inc. proprietary & confidential Time-bound exploration of

    technical or design solutions to address feasibility and help answer build, borrow, or buy decisions Spike
  25. © 2021 Productboard, Inc. proprietary & confidential Workshop

  26. © 2021 Productboard, Inc. proprietary & confidential We're a three-person

    team creating a B2C food ordering service for family members that are taking care of elderly parents. These family members may be located in a different location than their family members. They are interested in finding ways to ensure folks like their parents/grandparents have enough food and are eating healthy. We know they have lots of in-market options — grocery stores, UBER Eats, meal services that they can use. To address this problem we’ve built a mobile app where people can order on behalf of their family members -- managing everything from ordering to payments. How would we validate that our solution is valuable? https://miro.com/app/board/o9 J_lPm1BEM=/ Scenario
  27. © 2021 Productboard, Inc. proprietary & confidential Align on what

    you want to validate Step 1: Priorities
  28. © 2021 Productboard, Inc. proprietary & confidential Make a list

    of the most important things you want to validate Rate each 1-5 for importance (5 = critical, 1 = not very important) Rate each 1-5 for doubt (5 = most confident, 1 = least confident) Multiple each and write the result -- do those with the highest score Write out a short summary of what you want to know
  29. © 2021 Productboard, Inc. proprietary & confidential We believe {this

    capability} Will result in {this outcome} We will have confidence to process when we see {this measurable signal} State your hypothesis. Write it down. Step 2: Hypothesis
  30. © 2021 Productboard, Inc. proprietary & confidential Think: average, ratio,

    rates, percentage X% increase in customers that {take the desired action} Average # of customers were able to {complete the primary action} Ratio of customers were willing to buy vs. not What’s a measurable signal? Metrics
  31. © 2021 Productboard, Inc. proprietary & confidential Pick your approach(es),

    design your experiment(s) Step 3: How
  32. © 2021 Productboard, Inc. proprietary & confidential Determine who will

    be involved and what roles they will play Step 4: Who
  33. © 2021 Productboard, Inc. proprietary & confidential Get clear on

    when you need to do this by Step 5: When
  34. © 2021 Productboard, Inc. proprietary & confidential Do a test

    run Step 6: Practice
  35. © 2021 Productboard, Inc. proprietary & confidential Let’s do it

    Step 7: Execute
  36. © 2021 Productboard, Inc. proprietary & confidential Evaluate and summarize

    Step 8: Analyze
  37. © 2021 Productboard, Inc. proprietary & confidential Share, determine next

    steps, rinse and repeat Step 9: Share
  38. © 2021 Productboard, Inc. proprietary & confidential Q&A

  39. © 2021 Productboard, Inc. proprietary & confidential Toolkit

  40. © 2021 Productboard, Inc. proprietary & confidential • Surveys •

    Microsurveys • Clickable prototype • Single feature MVP • Mash up • Spike • Landing page • Physical prototype • Wizard of Oz • Split test • Presale • Letter of intent • Pop up • Crowdfunding Choose from any of these Methods for validation
  41. © 2021 Productboard, Inc. proprietary & confidential Asking better questions

    Question type Purpose Examples Sample Open Learn What, how, who, where, why What areas of your product process do you think needs the most improvement? Why hasn’t that change moved forward? Closed Confirm understanding Do, So, Is, Are Do you think what you’re doing today works? Are you finding that approach is helpful? Closed Seek commitment If, Can, Will, Would, Should, Could Can we do this by next week? Will you be able to meet on Tuesday to over our next steps?
  42. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Brainstorm questions • Write the questions • Edit and order • Write instructions • Test it • Field it & invite • Analyze Pros/Cons: • Cheap and easy to execute • Need to get respondents Multi-question survey used to learn Surveys
  43. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Grab a tool to do this for you (e.g. Examples: https://userpilot.com or https://userleap.com) or roll your own Pros/cons: • Collect a lot • High completion/submission rate • Can be targeted (e.g. based on product usage) Short survey shown to users of your product Microsurveys
  44. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Define your goals and script • Build your prototype • Test it internally • Walk customers through the script and take notes • Summarize key learnings Pros/cons: • Easy to build and validate key jobs • Can rely on a lot of tools to help Digital UI with clickable zones to represent the software’s reactions to the customer’s interactions Clickable prototype
  45. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Design the smallest version of a feature the solves a high-impact need/job • Test it internally • Get customers to use it • Gather feedback Pros/cons: • Focussed feedback on single need/job Functioning MVP with one working feature you need to validate Single feature MVP
  46. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Map out your flow • Find existing tools to integrate that complete the process/workflow • Integrate • Test • Have customers use it and gather feedback Pros/cons: • Increasingly lots of options here Functioning MVP that combines existing services to illustrate the value Mash up
  47. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Define acceptance criteria • Define time box • Undertake work • Analyze Pros/Cons: • Time bound go/no-go decisions • Takes team off other work Time-bound exploration of technical or design solutions to address feasibility and help answer build, borrow, or buy decisions Spike
  48. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Domain name • Build page - value proposition, CTA (email signup), customer pains, your solution • Integrate analytics • Drive traffic to your page • Analyze (viewed, signed up, spent time, engaged with content) Pros/Cons: • Relatively inexpensive • Not good for all products Simple web page with value proposition and CTA Landing page
  49. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Determine key job you want to test • Create physical example that substitutes • Try it • Validate learnings Pros/cons: • Both cassette tapes and iPhone used this approach (block of wood) • May not address key job you have for your product Something that substitutes for the actual physical product and helps you validate a particular use case Physical prototype
  50. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Plan the steps • Test the steps • Conduct the experiment • Analyze learnings Pros/Cons • Great way to test experiences • Can be labour intensive and require more setup time Deliver a customer experience manually with people rather than technology Wizard of Oz
  51. © 2021 Productboard, Inc. proprietary & confidential How to do

    it • Identify behaviour you want to test • Create control and variant • Identify sample size and confidence level • Run test (50% of traffic to each Pros/Cons: • Inexpensive • Traffic requirements for statistical significance not always possible Comparison of two versions, control A against variant B, to see which performs better Split test
  52. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Create landing page • Insert options • Make page available and drive traffic • Encourage people to buy before it’s available • Analyze (which options resonated?) Pros/Cons: • Sale of an item before it’s available for purchase Presale
  53. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Create your letter of intent • Show letter of intent to target audience • See if they’ll signup • Take notes (quotes, pains, needs, body language) • Review notes Pros/cons: • A good way to find customers who may be willing to continue and help you push ideas forward Short not legally binding contract Letter of intent
  54. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Find location • Design experience • Promote • Run store • Gather feedback Pros/Cons: • Can be a great way to validate physical products resonate with customers • Costs involved (permits, leasing, etc.) Temporary retail store where you sell your goods Pop up
  55. © 2021 Productboard, Inc. proprietary & confidential How to do

    it: • Define needs • Choose platform • Create pitch (video, text) of value proposition, CTA, customer pains and your solution, and pledge amounts • Make public • Drive traffic Pros/Cons: • Instant validation • Need to be ready to deliver Fund a project by raising many small amounts of money from a large group of people Crowdfunding