From Vanity to Results What’s the story with metrics? Aleksi Rossi @ Lean Startup Circle Wien 2013-07-26 

The  Open  Ministry   Open ministry Approved by Communication officials Equal marriage Dismantle alcohol monopoly Fur-farming ban Copyright law reform Government debt ceiling Energy drink prohibition for under 15 years 

No content

No content

is a vanity metric? is a good metric? kind of successes have you had with metrics? would you like to do better in your startup? What 

Lean Startup Build Product Measure Data Learn Ideas Minimum Viable Product Continuous deployment Scientific experiment Customer development Eliminate waste Split testing 5 Whys 

Methodology Lean Startup 

Methodology of maximizing probability of start-up success Lean Startup 

Methodology of maximizing probability of start-up success by eliminating waste Lean Startup 

Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers Lean Startup 

Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Lean Startup 

Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Lean Startup 

Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively Lean Startup 

Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption Lean Startup 

Lean Startup Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. 

Lean Startup Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky 

Lean Startup Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky, you’ll have ”product/market fit” 

Lean Startup Methodology of maximizing probability of start-up success by eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky, you’ll have ”product/market fit” and can concentrate on optimizing and scaling. 

No content

Good metric gives answers! No, that’s vanity! 

How to help with the dirty laundry? Build Product Measure Data Learn Ideas 

Week Users 1 1111 2 2666 3 3554 Vanity metric 0   500   1000   1500   2000   2500   3000   3500   4000   1   2   3   

Less Vanity metric Week Users Signed in Weekly active users (WAU) 1 1111 1111 100% 2 2666 1888 71% 3 3554 1466 41% 0%   25%   50%   75%   100%   1   2   3   Weekly  ac)ve  users   

Story so far: A new feature X was added and announced How do you know the new feature is any good? 

Week Users Signed up Signed in Used feature X Usefulness 1 1111 1111 1111 666 60% 2 2666 1555 1888 444 24% 3 3554 888 1466 333 23% Is the new feature useful? 0%   25%   50%   75%   100%   1   2   3   Usefulness   No! It should be removed! 

However, cohorts might tell a different story! Features are useful only if they deliver value if they are used! 

Cohort is a similar group of people. In this example, a group of people who started using at the same week. Week : 1 2 3 Cohort from week Signed up Signed in Used feature X Useful- ness Signed in Used feature X Useful- ness Signed in Used feature X Useful- ness 1 1111 1111 666 60% 333 167 50% 111 72 65% 2 1555 1555 277 18% 467 70 15% 3 888 888 191 21% Sums: 3554 1111 666 1888 444 1466 333 

Let’s simplify! Week : 1 2 3 Cohort from week Usefulness Usefulness Usefulness 1 60% 50% 65% 2 18% 15% 3 21% 

Week : 1 2 3 Cohort from week Usefulness Usefulness Usefulness 1 60% 50% 65% 2 18% 15% 3 21% 0%   25%   50%   75%   100%   1   2   3   Usefulness   All  users   Cohort  week  1   Is the new feature useful? Yes! First users use it! 

Week : 1 2 3 Cohort from week Usefulness Usefulness Usefulness 1 60% 50% 65% 2 18% 15% 3 21% Is the new feature useful? No! New users don’t find it! 0%   25%   50%   75%   100%   1   2   3   Usefulness   All  users   Cohort  week  1   Cohort  week  2   Cohort  Week  3   

What could help? Announcing was effective Teach them Listen! New users don’t find it! 

We just learned a thing! Build Product Measure Data Learn Ideas 

Increase skillset and toolset by introducing a Data Scientist! How to come up with the data in Lean startup? 

Common pitfalls of metrics Counts Counts of unimportant things Measuring wrong things No baseline, no comparison Too many metrics Measured for fun, not to remove the pain 

Clinic  A  is  killing  45%  more   Simpson's   paradox   Treatment  successes   PorBons   Clinic  A   Hospital  B   Clinic  A   Hospital  B   Dead   26   9   Dead   2.6%   1.8%   Survived   974   492   Survived   97.4%   98.2%   Total   1000   501   Total   100.0%   100.0%   

Cancer  –  Stage  A   Treatment  successes   PorBons   Clinic  A   Hospital  B   Clinic  A   Hospital  B   Dead   1   1   Dead   0.2%   0.3%   Survived   499   399   Survived   99.8%   99.8%   Total   500   400   Total   100.0%   100.0%   Cancer  –  Stage  B   Clinic  A   Klinikka  B   Clinic  A   Hospital  B   Dead   25   8   Dead   5.0%   7.9%   Survived   475   93   Survived   95.0%   92.1%   Total   500   101   Total   100.0%   100.0%   Hospital  B  is  killing     50%  more  on  Stage  A  and   Over  50%  more  on  Stage  B     

Simpson's   paradox   Treatment  successes   PorBons   Clinic  A   Hospital  B   Clinic  A   Hospital  B   Dead   26   9   Dead   2.6%   1.8%   Survived   974   492   Survived   97.4%   98.2%   Total   1000   501   Total   100.0%   100.0%   Cancer  –  Stage  A   Treatment  successes   PorBons   Clinic  A   Hospital  B   Clinic  A   Hospital  B   Dead   1   1   Dead   0.2%   0.3%   Survived   499   399   Survived   99.8%   99.8%   Total   500   400   Total   100.0%   100.0%   Cancer  –  Stage  B   Clinic  A   Klinikka  B   Clinic  A   Hospital  B   Dead   25   8   Dead   5.0%   7.9%   Survived   475   93   Survived   95.0%   92.1%   Total   500   101   Total   100.0%   100.0%   Hospital  B  is   killing     50%  or  more   Clinic  A  is   killing     50%  more   

Test case: How do you know you improved registration process? Possible metrics: 1)  Registrations are up 2)  More activity on the main functions 3)  Less page loads 4)  Less loss of people between ”Register” and registered 5)  Less %-loss between ”Register” and registered 6)  Less average time between Register and registered 7)  People report happy onboarding / Less compaints 

Time for Questions! 

Aleksi.Rossi@iki.fi @AlekRossi http://aleksirossi.com https://speakerdeck.com/rossi/LSC-Vienna-Metrics Thank you! 

http:// www.flickr.com/ photos/ visualpanic/ 2823427263/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/kikasz/ 6186223885/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/baboon/ 115446241/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ ricksflicks/ 5348698725/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ jornidzerda/ 4935388653/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ visualpanic/ 3236219002/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ le_plochingen/ 6595415697/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ miuenski/ 5394654161/ http:// www.flickr.com/ photos/myxi/ 4327438430/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ 29393867@N07/ 4286828753/ sizes/l/in/ photostream/