Share
Embed
Transcript
41 slides
-
From Vanity to Results What’s the story with metrics? Aleksi
Rossi @ Lean Startup Circle Wien 2013-07-26 -
The Open Ministry Open ministry Approved by Communication officials
Equal marriage Dismantle alcohol monopoly Fur-farming ban Copyright law reform Government debt ceiling Energy drink prohibition for under 15 years -
is a vanity metric? is a good metric? kind of
successes have you had with metrics? would you like to do better in your startup? What -
Lean Startup Build Product Measure Data Learn Ideas Minimum Viable
Product Continuous deployment Scientific experiment Customer development Eliminate waste Split testing 5 Whys -
Methodology of maximizing probability of start-up success by eliminating waste
by repeatedly finding the right idea by talking to customers Lean Startup -
Methodology of maximizing probability of start-up success by eliminating waste
by repeatedly finding the right idea by talking to customers and honing it into a vision. Lean Startup -
Methodology of maximizing probability of start-up success by eliminating waste
by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Lean Startup -
Methodology of maximizing probability of start-up success by eliminating waste
by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively Lean Startup -
Methodology of maximizing probability of start-up success by eliminating waste
by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption Lean Startup -
Lean Startup Methodology of maximizing probability of start-up success by
eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. -
Lean Startup Methodology of maximizing probability of start-up success by
eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky -
Lean Startup Methodology of maximizing probability of start-up success by
eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky, you’ll have ”product/market fit” -
Lean Startup Methodology of maximizing probability of start-up success by
eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky, you’ll have ”product/market fit” and can concentrate on optimizing and scaling. -
How to help with the dirty laundry? Build Product Measure
Data Learn Ideas -
Week Users 1 1111 2 2666 3 3554 Vanity metric
0 500 1000 1500 2000 2500 3000 3500 4000 1 2 3 -
Less Vanity metric Week Users Signed in Weekly active users
(WAU) 1 1111 1111 100% 2 2666 1888 71% 3 3554 1466 41% 0% 25% 50% 75% 100% 1 2 3 Weekly ac)ve users -
Story so far: A new feature X was added and
announced How do you know the new feature is any good? -
Week Users Signed up Signed in Used feature X Usefulness
1 1111 1111 1111 666 60% 2 2666 1555 1888 444 24% 3 3554 888 1466 333 23% Is the new feature useful? 0% 25% 50% 75% 100% 1 2 3 Usefulness No! It should be removed! -
However, cohorts might tell a different story! Features are useful
only if they deliver value if they are used! -
Cohort is a similar group of people. In this example,
a group of people who started using at the same week. Week : 1 2 3 Cohort from week Signed up Signed in Used feature X Useful- ness Signed in Used feature X Useful- ness Signed in Used feature X Useful- ness 1 1111 1111 666 60% 333 167 50% 111 72 65% 2 1555 1555 277 18% 467 70 15% 3 888 888 191 21% Sums: 3554 1111 666 1888 444 1466 333 -
Let’s simplify! Week : 1 2 3 Cohort from week
Usefulness Usefulness Usefulness 1 60% 50% 65% 2 18% 15% 3 21% -
Week : 1 2 3 Cohort from week Usefulness Usefulness
Usefulness 1 60% 50% 65% 2 18% 15% 3 21% 0% 25% 50% 75% 100% 1 2 3 Usefulness All users Cohort week 1 Is the new feature useful? Yes! First users use it! -
Week : 1 2 3 Cohort from week Usefulness Usefulness
Usefulness 1 60% 50% 65% 2 18% 15% 3 21% Is the new feature useful? No! New users don’t find it! 0% 25% 50% 75% 100% 1 2 3 Usefulness All users Cohort week 1 Cohort week 2 Cohort Week 3 -
What could help? Announcing was effective Teach them Listen! New
users don’t find it! -
Increase skillset and toolset by introducing a Data Scientist! How
to come up with the data in Lean startup? -
Common pitfalls of metrics Counts Counts of unimportant things Measuring
wrong things No baseline, no comparison Too many metrics Measured for fun, not to remove the pain -
Clinic A is killing 45% more Simpson's paradox
Treatment successes PorBons Clinic A Hospital B Clinic A Hospital B Dead 26 9 Dead 2.6% 1.8% Survived 974 492 Survived 97.4% 98.2% Total 1000 501 Total 100.0% 100.0% -
Cancer – Stage A Treatment successes PorBons
Clinic A Hospital B Clinic A Hospital B Dead 1 1 Dead 0.2% 0.3% Survived 499 399 Survived 99.8% 99.8% Total 500 400 Total 100.0% 100.0% Cancer – Stage B Clinic A Klinikka B Clinic A Hospital B Dead 25 8 Dead 5.0% 7.9% Survived 475 93 Survived 95.0% 92.1% Total 500 101 Total 100.0% 100.0% Hospital B is killing 50% more on Stage A and Over 50% more on Stage B -
Simpson's paradox Treatment successes PorBons Clinic
A Hospital B Clinic A Hospital B Dead 26 9 Dead 2.6% 1.8% Survived 974 492 Survived 97.4% 98.2% Total 1000 501 Total 100.0% 100.0% Cancer – Stage A Treatment successes PorBons Clinic A Hospital B Clinic A Hospital B Dead 1 1 Dead 0.2% 0.3% Survived 499 399 Survived 99.8% 99.8% Total 500 400 Total 100.0% 100.0% Cancer – Stage B Clinic A Klinikka B Clinic A Hospital B Dead 25 8 Dead 5.0% 7.9% Survived 475 93 Survived 95.0% 92.1% Total 500 101 Total 100.0% 100.0% Hospital B is killing 50% or more Clinic A is killing 50% more -
Test case: How do you know you improved registration process?
Possible metrics: 1) Registrations are up 2) More activity on the main functions 3) Less page loads 4) Less loss of people between ”Register” and registered 5) Less %-loss between ”Register” and registered 6) Less average time between Register and registered 7) People report happy onboarding / Less compaints -
http:// www.flickr.com/ photos/ visualpanic/ 2823427263/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/kikasz/
6186223885/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/baboon/ 115446241/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ ricksflicks/ 5348698725/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ jornidzerda/ 4935388653/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ visualpanic/ 3236219002/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ le_plochingen/ 6595415697/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ miuenski/ 5394654161/ http:// www.flickr.com/ photos/myxi/ 4327438430/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ 29393867@N07/ 4286828753/ sizes/l/in/ photostream/