Solution Validation and Metrics - UBC PM Club

© 2021 Productboard, Inc. proprietary & conﬁdential
Solution Validation (and Metrics)
UBC Product Management Club

View Slide

i
Product Excellence Consultant
[email protected]
Scott Baldwin

View Slide

Former customer and long-time
product leader
About Scott
🇨🇦 Based in Vancouver, BC, Canada
😀 20+ years experience as a PM and previously in
UX research & IA
👨🏻‍💻 Led product teams in a variety of companies —
Thinkiﬁc, Finning Digital, Central 1, Inovera Solutions,
Blue Zone Entertainment
🤝 Active participant in the product community –
passionate about coaching product teams and
mentoring those looking to enter PM and grow in
their careers
📚 Featured in the O’Reilly book Design Leadership
🥁 Music, ultramarathons 🏃🏻‍♂, and family

View Slide

1. Why we need to validate
2. Tenets of good validation practices
3. Methods/techniques and how to
choose the right one
4. Workshop
5. Q&A
6. Toolkit
Does anyone even pay
attention to these agenda
slides?
Agenda 📒

View Slide

🙋🏽‍♂ Why validate?
Go to www.slido.com and enter code #90055 to share your thoughts

View Slide

ⓘ Start presenting to display the poll results on this slide.
Why should we do solution validation?

View Slide

People = who
Problem = why
Product = how
Whoa, he’s already making
assumptions?
My assumptions
🧐

View Slide

Most ideas fail because...
High failure rate

View Slide

Valuable = will customers buy this or choose
to use it?
Usable = can the user ﬁgure out how to use
this?
Feasible = can we build this given the time,
skills, and technology we have?
Viable = does this work for the various
aspects of our business
Valuable, usable, feasible, and
viable
Four risks

View Slide

Build
Measure
Learn
prototype
data
ideas

View Slide

Guidelines

View Slide

Think in experiments, not projects

View Slide

DE&I

View Slide

Fast, easy, and eﬀective

View Slide

Start together, not alone

View Slide

Get out of the building

View Slide

🤨 Choosing the right method

View Slide

Which of the four risks are you validating
for?
What type of hypothesis are you testing?
How much evidence do you already have
for the speciﬁc hypothesis?
How much time do you have until the next
major decision point or until you run out of
money?
How will you know if you’re successful or
not?
Before you start
Questions to ask

View Slide

Reduce uncertainty as much as possible
before you build anything
Cheap and fast early in journey
Use multiple touchpoints to increase the
strength and likelihood of gathering
evidence
Pick your approach wisely - strongest
evidence given constraints that your team
can support
Keep in mind

View Slide

Methods/techniques 🤠

View Slide

Time
Strength of evidence
Clickable prototype
Single feature MVP
Mash up
Concierge
Landing page
Crowdfunding
Split test Presale
Survey
Wizard of Oz
Mock sale
Letter of intent
Pop up store
Spike
Usability testing
Guerilla testing

View Slide

Multi-question survey used to
learn
Surveys

View Slide

Digital UI with clickable zones
to represent the software’s
reactions to the customer’s
interactions
Clickable
prototype

View Slide

Time-bound exploration of
technical or design solutions to
address feasibility and help
answer build, borrow, or buy
decisions
Spike

View Slide

Workshop

View Slide

We're a three-person team creating a B2C food ordering
service for family members that are taking care of elderly
parents.
These family members may be located in a diﬀerent
location than their family members. They are interested in
ﬁnding ways to ensure folks like their parents/grandparents
have enough food and are eating healthy.
We know they have lots of in-market options — grocery
stores, UBER Eats, meal services that they can use.
To address this problem we’ve built a mobile app where
people can order on behalf of their family members --
managing everything from ordering to payments.
How would we validate that our solution is valuable?
https://miro.com/app/board/o9
J_lPm1BEM=/
Scenario

View Slide

Align on what you want to
validate
Step 1: Priorities

View Slide

Make a list of the most
important things you want to
validate
Rate each 1-5 for importance
(5 = critical, 1 = not very
important)
Rate each 1-5 for doubt (5 =
most conﬁdent, 1 = least
conﬁdent)
Multiple each and write the
result -- do those with the
highest score
Write out a short summary of
what you want to know

View Slide

We believe {this capability}
Will result in {this outcome}
We will have conﬁdence to
process when we see {this
measurable signal}
State your hypothesis. Write it
down.
Step 2: Hypothesis

View Slide

Think: average, ratio, rates, percentage
X% increase in customers that {take the
desired action}
Average # of customers were able to
{complete the primary action}
Ratio of customers were willing to buy vs.
not
What’s a measurable signal?
Metrics

View Slide

Pick your approach(es), design
your experiment(s)
Step 3: How

View Slide

Determine who will be involved
and what roles they will play
Step 4: Who

View Slide

Get clear on when you need to
do this by
Step 5: When

View Slide

Do a test run
Step 6: Practice

View Slide

Let’s do it
Step 7: Execute

View Slide

Evaluate and summarize
Step 8: Analyze

View Slide

Share, determine next steps,
rinse and repeat
Step 9: Share

View Slide

Q&A

View Slide

Toolkit

View Slide

● Surveys
● Microsurveys
● Clickable prototype
● Single feature MVP
● Mash up
● Spike
● Landing page
● Physical prototype
● Wizard of Oz
● Split test
● Presale
● Letter of intent
● Pop up
● Crowdfunding
Choose from any of these
Methods for
validation

View Slide

Asking better questions
Question type Purpose Examples Sample
Open Learn What, how, who, where, why What areas of your product
process do you think needs the
most improvement?
Why hasn’t that change moved
forward?
Closed Conﬁrm understanding Do, So, Is, Are Do you think what you’re doing
today works?
Are you ﬁnding that approach is
helpful?
Closed Seek commitment If, Can, Will, Would, Should,
Could
Can we do this by next week?
Will you be able to meet on
Tuesday to over our next steps?

View Slide

How to do it:
● Brainstorm questions
● Write the questions
● Edit and order
● Write instructions
● Test it
● Field it & invite
● Analyze
Pros/Cons:
● Cheap and easy to execute
● Need to get respondents
Multi-question survey used to
learn
Surveys

View Slide

How to do it:
● Grab a tool to do this for you (e.g.
Examples: https://userpilot.com or
https://userleap.com) or roll your own
Pros/cons:
● Collect a lot
● High completion/submission rate
● Can be targeted (e.g. based on
product usage)
Short survey shown to users of
your product
Microsurveys

View Slide

How to do it:
● Deﬁne your goals and script
● Build your prototype
● Test it internally
● Walk customers through the script
and take notes
● Summarize key learnings
Pros/cons:
● Easy to build and validate key jobs
● Can rely on a lot of tools to help
Digital UI with clickable zones
to represent the software’s
reactions to the customer’s
interactions
Clickable
prototype

View Slide

How to do it:
● Design the smallest version of a
feature the solves a high-impact
need/job
● Test it internally
● Get customers to use it
● Gather feedback
Pros/cons:
● Focussed feedback on single
need/job
Functioning MVP with one
working feature you need to
validate
Single feature
MVP

View Slide

How to do it:
● Map out your ﬂow
● Find existing tools to integrate that
complete the process/workﬂow
● Integrate
● Test
● Have customers use it and gather
feedback
Pros/cons:
● Increasingly lots of options here
Functioning MVP that
combines existing services to
illustrate the value
Mash up

View Slide

How to do it:
● Define acceptance criteria
● Define time box
● Undertake work
● Analyze
Pros/Cons:
● Time bound go/no-go decisions
● Takes team off other work
Time-bound exploration of
technical or design solutions to
address feasibility and help
answer build, borrow, or buy
decisions
Spike

View Slide

How to do it:
● Domain name
● Build page - value proposition, CTA
(email signup), customer pains, your
solution
● Integrate analytics
● Drive traﬃc to your page
● Analyze (viewed, signed up, spent
time, engaged with content)
Pros/Cons:
● Relatively inexpensive
● Not good for all products
Simple web page with value
proposition and CTA
Landing page

View Slide

How to do it:
● Determine key job you want to test
● Create physical example that
substitutes
● Try it
● Validate learnings
Pros/cons:
● Both cassette tapes and iPhone used
this approach (block of wood)
● May not address key job you have for
your product
Something that substitutes for
the actual physical product and
helps you validate a particular
use case
Physical prototype

View Slide

How to do it:
● Plan the steps
● Test the steps
● Conduct the experiment
● Analyze learnings
Pros/Cons
● Great way to test experiences
● Can be labour intensive and require
more setup time
Deliver a customer experience
manually with people rather
than technology
Wizard of Oz

View Slide

How to do it
● Identify behaviour you want to test
● Create control and variant
● Identify sample size and confidence
level
● Run test (50% of traffic to each
Pros/Cons:
● Inexpensive
● Traffic requirements for statistical
significance not always possible
Comparison of two versions,
control A against variant B, to
see which performs better
Split test

View Slide

How to do it:
● Create landing page
● Insert options
● Make page available and drive traﬃc
● Encourage people to buy before it’s
available
● Analyze (which options resonated?)
Pros/Cons:
●
Sale of an item before it’s
available for purchase
Presale

View Slide

How to do it:
● Create your letter of intent
● Show letter of intent to target
audience
● See if they’ll signup
● Take notes (quotes, pains, needs,
body language)
● Review notes
Pros/cons:
● A good way to ﬁnd customers who
may be willing to continue and help
you push ideas forward
Short not legally binding
contract
Letter of intent

View Slide

How to do it:
● Find location
● Design experience
● Promote
● Run store
● Gather feedback
Pros/Cons:
● Can be a great way to validate
physical products resonate with
customers
● Costs involved (permits, leasing, etc.)
Temporary retail store where
you sell your goods
Pop up

View Slide

How to do it:
● Deﬁne needs
● Choose platform
● Create pitch (video, text) of value
proposition, CTA, customer pains and
your solution, and pledge amounts
● Make public
● Drive traﬃc
Pros/Cons:
● Instant validation
● Need to be ready to deliver
Fund a project by raising many
small amounts of money from a
large group of people
Crowdfunding

View Slide

Solution Validation and Metrics - UBC PM Club

Solution Validation and Metrics - UBC PM Club

Scott Baldwin

More Decks by Scott Baldwin

Other Decks in Technology

Featured

Transcript