Solution Validation and Metrics - UBC PM Club

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

© 2021 Productboard, Inc. proprietary & conﬁdential Former customer and long-time product leader About Scott 🇨🇦 Based in Vancouver, BC, Canada 😀 20+ years experience as a PM and previously in UX research & IA 👨🏻‍💻 Led product teams in a variety of companies — Thinkiﬁc, Finning Digital, Central 1, Inovera Solutions, Blue Zone Entertainment 🤝 Active participant in the product community – passionate about coaching product teams and mentoring those looking to enter PM and grow in their careers 📚 Featured in the O’Reilly book Design Leadership 🥁 Music, ultramarathons 🏃🏻‍♂, and family

Slide 4

Slide 4 text

© 2021 Productboard, Inc. proprietary & conﬁdential 1. Why we need to validate 2. Tenets of good validation practices 3. Methods/techniques and how to choose the right one 4. Workshop 5. Q&A 6. Toolkit Does anyone even pay attention to these agenda slides? Agenda 📒

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

© 2021 Productboard, Inc. proprietary & conﬁdential Valuable = will customers buy this or choose to use it? Usable = can the user ﬁgure out how to use this? Feasible = can we build this given the time, skills, and technology we have? Viable = does this work for the various aspects of our business Valuable, usable, feasible, and viable Four risks

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

© 2021 Productboard, Inc. proprietary & conﬁdential Which of the four risks are you validating for? What type of hypothesis are you testing? How much evidence do you already have for the speciﬁc hypothesis? How much time do you have until the next major decision point or until you run out of money? How will you know if you’re successful or not? Before you start Questions to ask

Slide 19

Slide 19 text

© 2021 Productboard, Inc. proprietary & conﬁdential Reduce uncertainty as much as possible before you build anything Cheap and fast early in journey Use multiple touchpoints to increase the strength and likelihood of gathering evidence Pick your approach wisely - strongest evidence given constraints that your team can support Keep in mind

Slide 20

Slide 20 text

Slide 21

Slide 21 text

© 2021 Productboard, Inc. proprietary & conﬁdential Time Strength of evidence Clickable prototype Single feature MVP Mash up Concierge Landing page Crowdfunding Split test Presale Survey Wizard of Oz Mock sale Letter of intent Pop up store Spike Usability testing Guerilla testing

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

© 2021 Productboard, Inc. proprietary & confidential We're a three-person team creating a B2C food ordering service for family members that are taking care of elderly parents. These family members may be located in a different location than their family members. They are interested in finding ways to ensure folks like their parents/grandparents have enough food and are eating healthy. We know they have lots of in-market options — grocery stores, UBER Eats, meal services that they can use. To address this problem we’ve built a mobile app where people can order on behalf of their family members -- managing everything from ordering to payments. How would we validate that our solution is valuable? https://miro.com/app/board/o9 J_lPm1BEM=/ Scenario

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2021 Productboard, Inc. proprietary & confidential Make a list of the most important things you want to validate Rate each 1-5 for importance (5 = critical, 1 = not very important) Rate each 1-5 for doubt (5 = most confident, 1 = least confident) Multiple each and write the result -- do those with the highest score Write out a short summary of what you want to know

Slide 29

Slide 29 text

© 2021 Productboard, Inc. proprietary & conﬁdential We believe {this capability} Will result in {this outcome} We will have conﬁdence to process when we see {this measurable signal} State your hypothesis. Write it down. Step 2: Hypothesis

Slide 30

Slide 30 text

© 2021 Productboard, Inc. proprietary & conﬁdential Think: average, ratio, rates, percentage X% increase in customers that {take the desired action} Average # of customers were able to {complete the primary action} Ratio of customers were willing to buy vs. not What’s a measurable signal? Metrics

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

© 2021 Productboard, Inc. proprietary & conﬁdential ● Surveys ● Microsurveys ● Clickable prototype ● Single feature MVP ● Mash up ● Spike ● Landing page ● Physical prototype ● Wizard of Oz ● Split test ● Presale ● Letter of intent ● Pop up ● Crowdfunding Choose from any of these Methods for validation

Slide 41

Slide 41 text

© 2021 Productboard, Inc. proprietary & confidential Asking better questions Question type Purpose Examples Sample Open Learn What, how, who, where, why What areas of your product process do you think needs the most improvement? Why hasn’t that change moved forward? Closed Confirm understanding Do, So, Is, Are Do you think what you’re doing today works? Are you finding that approach is helpful? Closed Seek commitment If, Can, Will, Would, Should, Could Can we do this by next week? Will you be able to meet on Tuesday to over our next steps?

Slide 42

Slide 42 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Brainstorm questions ● Write the questions ● Edit and order ● Write instructions ● Test it ● Field it & invite ● Analyze Pros/Cons: ● Cheap and easy to execute ● Need to get respondents Multi-question survey used to learn Surveys

Slide 43

Slide 43 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Grab a tool to do this for you (e.g. Examples: https://userpilot.com or https://userleap.com) or roll your own Pros/cons: ● Collect a lot ● High completion/submission rate ● Can be targeted (e.g. based on product usage) Short survey shown to users of your product Microsurveys

Slide 44

Slide 44 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Deﬁne your goals and script ● Build your prototype ● Test it internally ● Walk customers through the script and take notes ● Summarize key learnings Pros/cons: ● Easy to build and validate key jobs ● Can rely on a lot of tools to help Digital UI with clickable zones to represent the software’s reactions to the customer’s interactions Clickable prototype

Slide 45

Slide 45 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Design the smallest version of a feature the solves a high-impact need/job ● Test it internally ● Get customers to use it ● Gather feedback Pros/cons: ● Focussed feedback on single need/job Functioning MVP with one working feature you need to validate Single feature MVP

Slide 46

Slide 46 text

© 2021 Productboard, Inc. proprietary & confidential How to do it: ● Map out your flow ● Find existing tools to integrate that complete the process/workflow ● Integrate ● Test ● Have customers use it and gather feedback Pros/cons: ● Increasingly lots of options here Functioning MVP that combines existing services to illustrate the value Mash up

Slide 47

Slide 47 text

© 2021 Productboard, Inc. proprietary & confidential How to do it: ● Define acceptance criteria ● Define time box ● Undertake work ● Analyze Pros/Cons: ● Time bound go/no-go decisions ● Takes team off other work Time-bound exploration of technical or design solutions to address feasibility and help answer build, borrow, or buy decisions Spike

Slide 48

Slide 48 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Domain name ● Build page - value proposition, CTA (email signup), customer pains, your solution ● Integrate analytics ● Drive traﬃc to your page ● Analyze (viewed, signed up, spent time, engaged with content) Pros/Cons: ● Relatively inexpensive ● Not good for all products Simple web page with value proposition and CTA Landing page

Slide 49

Slide 49 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Determine key job you want to test ● Create physical example that substitutes ● Try it ● Validate learnings Pros/cons: ● Both cassette tapes and iPhone used this approach (block of wood) ● May not address key job you have for your product Something that substitutes for the actual physical product and helps you validate a particular use case Physical prototype

Slide 50

Slide 50 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Plan the steps ● Test the steps ● Conduct the experiment ● Analyze learnings Pros/Cons ● Great way to test experiences ● Can be labour intensive and require more setup time Deliver a customer experience manually with people rather than technology Wizard of Oz

Slide 51

Slide 51 text

© 2021 Productboard, Inc. proprietary & confidential How to do it ● Identify behaviour you want to test ● Create control and variant ● Identify sample size and confidence level ● Run test (50% of traffic to each Pros/Cons: ● Inexpensive ● Traffic requirements for statistical significance not always possible Comparison of two versions, control A against variant B, to see which performs better Split test

Slide 52

Slide 52 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Create landing page ● Insert options ● Make page available and drive traﬃc ● Encourage people to buy before it’s available ● Analyze (which options resonated?) Pros/Cons: ● Sale of an item before it’s available for purchase Presale

Slide 53

Slide 53 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Create your letter of intent ● Show letter of intent to target audience ● See if they’ll signup ● Take notes (quotes, pains, needs, body language) ● Review notes Pros/cons: ● A good way to ﬁnd customers who may be willing to continue and help you push ideas forward Short not legally binding contract Letter of intent

Slide 54

Slide 54 text

© 2021 Productboard, Inc. proprietary & conﬁdential How to do it: ● Find location ● Design experience ● Promote ● Run store ● Gather feedback Pros/Cons: ● Can be a great way to validate physical products resonate with customers ● Costs involved (permits, leasing, etc.) Temporary retail store where you sell your goods Pop up

Slide 55

Slide 55 text

© 2021 Productboard, Inc. proprietary & confidential How to do it: ● Define needs ● Choose platform ● Create pitch (video, text) of value proposition, CTA, customer pains and your solution, and pledge amounts ● Make public ● Drive traffic Pros/Cons: ● Instant validation ● Need to be ready to deliver Fund a project by raising many small amounts of money from a large group of people Crowdfunding