Metrics-Driven Development: See the forest for the trees

Metrics-Driven Development: See the forest for the trees

UPDATED SLIDES: October 2015 - GitHub Universe

Talk originally given at Open Data Science in Boston, May 2015 and EuroPython in July 2015.

Full blog post: http://rogue.ly/metrics

8c5e76dca74a59822dbf7f0286177ddd?s=128

Lynn Root

May 30, 2015
Tweet

Transcript

  1. 2.

    $ whoami • Backend Engineer @ Spotify, based in San

    Fransisco • Board Member of Python Software Foundation • Founder of PyLadies of San Francisco, one of the lead organizers of PyLadies globally
  2. 3.

    • About Spotify • Our agile approach to & implementation

    of metrics • Stepping back for the bigger picture See the forest for the trees.
  3. 6.

    About Spotify. • Available in 58 countries • Over 20

    million paying subscribers • Over 75 million monthly active users • Over 30 million songs; 20,000 added each day • Paid over $3 billion to rights holders to date • Main engineering offices located in Stockholm & New York 

  4. 7.

    About Spotify. • 30 TB of user & service-related log

    data per day • 400 TB of data generated in Hadoop each day • 1600+ node Hadoop cluster • 17,000 Hadoop jobs per day • 60 PBs of storage capacity
  5. 11.

    Feedback Expectations Customers Service/Product • What do we deliver? •

    For whom do we produce it? Who defines our work? Finding our Target Condition
  6. 12.

    Feedback Expectations Customers Service/Product • What do we deliver? •

    For whom do we produce it? Who defines our work? • What do they expect of us? Finding our Target Condition
  7. 13.

    Feedback Expectations Customers Service/Product • What do we deliver? •

    For whom do we produce it? Who defines our work? • What do they expect of us? • Did we meet those expectations? Finding our Target Condition
  8. 15.

    Metrics we wanted to see ‣ Signup/auth flow abandonment ‣

    Facebook-connected users – percentage of total users, trend over time ‣ Percent of users that sign up through Facebook per hour/day/ week ‣ Facebook-related errors
  9. 16.

    Metrics we wanted to see ‣ Daily Active Users by

    Partner/Feature ‣ Registrations, subscription rate, and referrals by Partner ‣ Web API usage by Partner ‣ Squad-focused Twitter feed ("uber + spotify," etc.) – what’s being complained about that neither the partner or we may not see? ‣ Outstanding JIRA issues ‣ Request count by internal requesting service/team
  10. 17.

    New Processes ‣ Evaluate metrics during retrospectives. ‣ Set metric

    goals to new integration projects. ‣ Soliciting feedback from external partners.
  11. 19.

    Creating a new metric ‣ How do metrics map to

    business goals? ‣ How do you prioritize different goals you want to drive? ‣ How can we create dashboards that are actually actionable?
  12. 20.

    Representing metrics ‣ How do we correctly measure what we

    care about? ‣ What’s best: a gauge, counter, meter, histogram, or a timer? ‣ What’s a better representation for a metric, the mean, 95% 99%?
  13. 21.

    Consuming metrics ‣ How often do you check in on

    metrics? ‣ How do you make dashboards more visible? ‣ Considerations for the setup?
  14. 22.

    Iterating over current metrics ‣ Why did we not reach

    our goal? ‣ What action are we going to take? ‣ What about unknowns?