Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LSC Vienna Metrics

Aleksi
July 26, 2013

LSC Vienna Metrics

Metrics prez @ Lean Startup Circle Vienna Meetup

Aleksi

July 26, 2013
Tweet

More Decks by Aleksi

Other Decks in Technology

Transcript

  1. From Vanity to Results What’s the story with metrics? Aleksi

    Rossi @ Lean Startup Circle Wien 2013-07-26
  2. The  Open  Ministry   Open ministry Approved by Communication officials

    Equal marriage Dismantle alcohol monopoly Fur-farming ban Copyright law reform Government debt ceiling Energy drink prohibition for under 15 years
  3. is a vanity metric? is a good metric? kind of

    successes have you had with metrics? would you like to do better in your startup? What
  4. Lean Startup Build Product Measure Data Learn Ideas Minimum Viable

    Product Continuous deployment Scientific experiment Customer development Eliminate waste Split testing 5 Whys
  5. Methodology of maximizing probability of start-up success by eliminating waste

    by repeatedly finding the right idea by talking to customers Lean Startup
  6. Methodology of maximizing probability of start-up success by eliminating waste

    by repeatedly finding the right idea by talking to customers and honing it into a vision. Lean Startup
  7. Methodology of maximizing probability of start-up success by eliminating waste

    by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Lean Startup
  8. Methodology of maximizing probability of start-up success by eliminating waste

    by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively Lean Startup
  9. Methodology of maximizing probability of start-up success by eliminating waste

    by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption Lean Startup
  10. Lean Startup Methodology of maximizing probability of start-up success by

    eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”.
  11. Lean Startup Methodology of maximizing probability of start-up success by

    eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky
  12. Lean Startup Methodology of maximizing probability of start-up success by

    eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky, you’ll have ”product/market fit”
  13. Lean Startup Methodology of maximizing probability of start-up success by

    eliminating waste by repeatedly finding the right idea by talking to customers and honing it into a vision. Avoiding assumption it’s the actually the idea that wins. Building a minimal product (/service) iteratively by defining riskiest assumption, and validating it by devising a ”scientific experiment”. Eventually, if you’re lucky, you’ll have ”product/market fit” and can concentrate on optimizing and scaling.
  14. Week Users 1 1111 2 2666 3 3554 Vanity metric

    0   500   1000   1500   2000   2500   3000   3500   4000   1   2   3  
  15. Less Vanity metric Week Users Signed in Weekly active users

    (WAU) 1 1111 1111 100% 2 2666 1888 71% 3 3554 1466 41% 0%   25%   50%   75%   100%   1   2   3   Weekly  ac)ve  users  
  16. Story so far: A new feature X was added and

    announced How do you know the new feature is any good?
  17. Week Users Signed up Signed in Used feature X Usefulness

    1 1111 1111 1111 666 60% 2 2666 1555 1888 444 24% 3 3554 888 1466 333 23% Is the new feature useful? 0%   25%   50%   75%   100%   1   2   3   Usefulness   No! It should be removed!
  18. However, cohorts might tell a different story! Features are useful

    only if they deliver value if they are used!
  19. Cohort is a similar group of people. In this example,

    a group of people who started using at the same week. Week : 1 2 3 Cohort from week Signed up Signed in Used feature X Useful- ness Signed in Used feature X Useful- ness Signed in Used feature X Useful- ness 1 1111 1111 666 60% 333 167 50% 111 72 65% 2 1555 1555 277 18% 467 70 15% 3 888 888 191 21% Sums: 3554 1111 666 1888 444 1466 333
  20. Let’s simplify! Week : 1 2 3 Cohort from week

    Usefulness Usefulness Usefulness 1 60% 50% 65% 2 18% 15% 3 21%
  21. Week : 1 2 3 Cohort from week Usefulness Usefulness

    Usefulness 1 60% 50% 65% 2 18% 15% 3 21% 0%   25%   50%   75%   100%   1   2   3   Usefulness   All  users   Cohort  week  1   Is the new feature useful? Yes! First users use it!
  22. Week : 1 2 3 Cohort from week Usefulness Usefulness

    Usefulness 1 60% 50% 65% 2 18% 15% 3 21% Is the new feature useful? No! New users don’t find it! 0%   25%   50%   75%   100%   1   2   3   Usefulness   All  users   Cohort  week  1   Cohort  week  2   Cohort  Week  3  
  23. Increase skillset and toolset by introducing a Data Scientist! How

    to come up with the data in Lean startup?
  24. Common pitfalls of metrics Counts Counts of unimportant things Measuring

    wrong things No baseline, no comparison Too many metrics Measured for fun, not to remove the pain
  25. Clinic  A  is  killing  45%  more   Simpson's   paradox

      Treatment  successes   PorBons   Clinic  A   Hospital  B   Clinic  A   Hospital  B   Dead   26   9   Dead   2.6%   1.8%   Survived   974   492   Survived   97.4%   98.2%   Total   1000   501   Total   100.0%   100.0%  
  26. Cancer  –  Stage  A   Treatment  successes   PorBons  

    Clinic  A   Hospital  B   Clinic  A   Hospital  B   Dead   1   1   Dead   0.2%   0.3%   Survived   499   399   Survived   99.8%   99.8%   Total   500   400   Total   100.0%   100.0%   Cancer  –  Stage  B   Clinic  A   Klinikka  B   Clinic  A   Hospital  B   Dead   25   8   Dead   5.0%   7.9%   Survived   475   93   Survived   95.0%   92.1%   Total   500   101   Total   100.0%   100.0%   Hospital  B  is  killing     50%  more  on  Stage  A  and   Over  50%  more  on  Stage  B    
  27. Simpson's   paradox   Treatment  successes   PorBons   Clinic

     A   Hospital  B   Clinic  A   Hospital  B   Dead   26   9   Dead   2.6%   1.8%   Survived   974   492   Survived   97.4%   98.2%   Total   1000   501   Total   100.0%   100.0%   Cancer  –  Stage  A   Treatment  successes   PorBons   Clinic  A   Hospital  B   Clinic  A   Hospital  B   Dead   1   1   Dead   0.2%   0.3%   Survived   499   399   Survived   99.8%   99.8%   Total   500   400   Total   100.0%   100.0%   Cancer  –  Stage  B   Clinic  A   Klinikka  B   Clinic  A   Hospital  B   Dead   25   8   Dead   5.0%   7.9%   Survived   475   93   Survived   95.0%   92.1%   Total   500   101   Total   100.0%   100.0%   Hospital  B  is   killing     50%  or  more   Clinic  A  is   killing     50%  more  
  28. Test case: How do you know you improved registration process?

    Possible metrics: 1)  Registrations are up 2)  More activity on the main functions 3)  Less page loads 4)  Less loss of people between ”Register” and registered 5)  Less %-loss between ”Register” and registered 6)  Less average time between Register and registered 7)  People report happy onboarding / Less compaints
  29. http:// www.flickr.com/ photos/ visualpanic/ 2823427263/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/kikasz/

    6186223885/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/baboon/ 115446241/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ ricksflicks/ 5348698725/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ jornidzerda/ 4935388653/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ visualpanic/ 3236219002/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ le_plochingen/ 6595415697/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ miuenski/ 5394654161/ http:// www.flickr.com/ photos/myxi/ 4327438430/ sizes/l/in/ photostream/ http:// www.flickr.com/ photos/ 29393867@N07/ 4286828753/ sizes/l/in/ photostream/