long-time product leader About Scott 🇨🇦 Based in Vancouver, BC, Canada 😀 20+ years experience as a PM and previously in UX research & IA 👨🏻💻 Led product teams in a variety of companies — Thinkific, Finning Digital, Central 1, Inovera Solutions, Blue Zone Entertainment 🤝 Active participant in the product community – passionate about coaching product teams and mentoring those looking to enter PM and grow in their careers 📚 Featured in the O’Reilly book Design Leadership 🥁 Music, ultramarathons 🏃🏻♂, and family
need to validate 2. Tenets of good validation practices 3. Methods/techniques and how to choose the right one 4. Workshop 5. Q&A 6. Toolkit Does anyone even pay attention to these agenda slides? Agenda 📒
customers buy this or choose to use it? Usable = can the user figure out how to use this? Feasible = can we build this given the time, skills, and technology we have? Viable = does this work for the various aspects of our business Valuable, usable, feasible, and viable Four risks
four risks are you validating for? What type of hypothesis are you testing? How much evidence do you already have for the specific hypothesis? How much time do you have until the next major decision point or until you run out of money? How will you know if you’re successful or not? Before you start Questions to ask
much as possible before you build anything Cheap and fast early in journey Use multiple touchpoints to increase the strength and likelihood of gathering evidence Pick your approach wisely - strongest evidence given constraints that your team can support Keep in mind
evidence Clickable prototype Single feature MVP Mash up Concierge Landing page Crowdfunding Split test Presale Survey Wizard of Oz Mock sale Letter of intent Pop up store Spike Usability testing Guerilla testing
team creating a B2C food ordering service for family members that are taking care of elderly parents. These family members may be located in a different location than their family members. They are interested in finding ways to ensure folks like their parents/grandparents have enough food and are eating healthy. We know they have lots of in-market options — grocery stores, UBER Eats, meal services that they can use. To address this problem we’ve built a mobile app where people can order on behalf of their family members -- managing everything from ordering to payments. How would we validate that our solution is valuable? https://miro.com/app/board/o9 J_lPm1BEM=/ Scenario
of the most important things you want to validate Rate each 1-5 for importance (5 = critical, 1 = not very important) Rate each 1-5 for doubt (5 = most confident, 1 = least confident) Multiple each and write the result -- do those with the highest score Write out a short summary of what you want to know
capability} Will result in {this outcome} We will have confidence to process when we see {this measurable signal} State your hypothesis. Write it down. Step 2: Hypothesis
rates, percentage X% increase in customers that {take the desired action} Average # of customers were able to {complete the primary action} Ratio of customers were willing to buy vs. not What’s a measurable signal? Metrics
Microsurveys • Clickable prototype • Single feature MVP • Mash up • Spike • Landing page • Physical prototype • Wizard of Oz • Split test • Presale • Letter of intent • Pop up • Crowdfunding Choose from any of these Methods for validation
Question type Purpose Examples Sample Open Learn What, how, who, where, why What areas of your product process do you think needs the most improvement? Why hasn’t that change moved forward? Closed Confirm understanding Do, So, Is, Are Do you think what you’re doing today works? Are you finding that approach is helpful? Closed Seek commitment If, Can, Will, Would, Should, Could Can we do this by next week? Will you be able to meet on Tuesday to over our next steps?
it: • Brainstorm questions • Write the questions • Edit and order • Write instructions • Test it • Field it & invite • Analyze Pros/Cons: • Cheap and easy to execute • Need to get respondents Multi-question survey used to learn Surveys
it: • Grab a tool to do this for you (e.g. Examples: https://userpilot.com or https://userleap.com) or roll your own Pros/cons: • Collect a lot • High completion/submission rate • Can be targeted (e.g. based on product usage) Short survey shown to users of your product Microsurveys
it: • Define your goals and script • Build your prototype • Test it internally • Walk customers through the script and take notes • Summarize key learnings Pros/cons: • Easy to build and validate key jobs • Can rely on a lot of tools to help Digital UI with clickable zones to represent the software’s reactions to the customer’s interactions Clickable prototype
it: • Design the smallest version of a feature the solves a high-impact need/job • Test it internally • Get customers to use it • Gather feedback Pros/cons: • Focussed feedback on single need/job Functioning MVP with one working feature you need to validate Single feature MVP
it: • Map out your flow • Find existing tools to integrate that complete the process/workflow • Integrate • Test • Have customers use it and gather feedback Pros/cons: • Increasingly lots of options here Functioning MVP that combines existing services to illustrate the value Mash up
it: • Define acceptance criteria • Define time box • Undertake work • Analyze Pros/Cons: • Time bound go/no-go decisions • Takes team off other work Time-bound exploration of technical or design solutions to address feasibility and help answer build, borrow, or buy decisions Spike
it: • Domain name • Build page - value proposition, CTA (email signup), customer pains, your solution • Integrate analytics • Drive traffic to your page • Analyze (viewed, signed up, spent time, engaged with content) Pros/Cons: • Relatively inexpensive • Not good for all products Simple web page with value proposition and CTA Landing page
it: • Determine key job you want to test • Create physical example that substitutes • Try it • Validate learnings Pros/cons: • Both cassette tapes and iPhone used this approach (block of wood) • May not address key job you have for your product Something that substitutes for the actual physical product and helps you validate a particular use case Physical prototype
it: • Plan the steps • Test the steps • Conduct the experiment • Analyze learnings Pros/Cons • Great way to test experiences • Can be labour intensive and require more setup time Deliver a customer experience manually with people rather than technology Wizard of Oz
it • Identify behaviour you want to test • Create control and variant • Identify sample size and confidence level • Run test (50% of traffic to each Pros/Cons: • Inexpensive • Traffic requirements for statistical significance not always possible Comparison of two versions, control A against variant B, to see which performs better Split test
it: • Create landing page • Insert options • Make page available and drive traffic • Encourage people to buy before it’s available • Analyze (which options resonated?) Pros/Cons: • Sale of an item before it’s available for purchase Presale
it: • Create your letter of intent • Show letter of intent to target audience • See if they’ll signup • Take notes (quotes, pains, needs, body language) • Review notes Pros/cons: • A good way to find customers who may be willing to continue and help you push ideas forward Short not legally binding contract Letter of intent
it: • Find location • Design experience • Promote • Run store • Gather feedback Pros/Cons: • Can be a great way to validate physical products resonate with customers • Costs involved (permits, leasing, etc.) Temporary retail store where you sell your goods Pop up
it: • Define needs • Choose platform • Create pitch (video, text) of value proposition, CTA, customer pains and your solution, and pledge amounts • Make public • Drive traffic Pros/Cons: • Instant validation • Need to be ready to deliver Fund a project by raising many small amounts of money from a large group of people Crowdfunding
benry
hirotomotaguchi
nenonaninu
nagisa53
mitchan
ishikawa_satoru
oracle4engineer
sat
zigorou
coosuke
kzsuzuki
z63d
ogaclejapan
mongodb
reverentgeek
aleyda
andyhume
addyosmani
sergeychernyshev
skipperchong
tammyeverts
panda_program
bcantrill
rasmusluckow
sugarenia
