Metrics-Driven Development: See the forest for the trees

UPDATED SLIDES: October 2015 - GitHub Universe

Talk originally given at Open Data Science in Boston, May 2015 and EuroPython in July 2015.

Full blog post: http://rogue.ly/metrics

Transcript

1. Metrics-Driven Development Lynn Root
 @roguelynn
2. $ whoami • Backend Engineer @ Spotify, based in San Fransisco • Board Member of Python Software Foundation • Founder of PyLadies of San Francisco, one of the lead organizers of PyLadies globally
3. • About Spotify • Our agile approach to & implementation of metrics • Stepping back for the bigger picture See the forest for the trees.
4. TL;DR: If it moves, you can track it.
 But should you?
5. About Spotify
6. About Spotify. • Available in 58 countries • Over 20 million paying subscribers • Over 75 million monthly active users • Over 30 million songs; 20,000 added each day • Paid over $3 billion to rights holders to date • Main engineering offices located in Stockholm & New York 

7. About Spotify. • 30 TB of user & service-related log data per day • 400 TB of data generated in Hadoop each day • 1600+ node Hadoop cluster • 17,000 Hadoop jobs per day • 60 PBs of storage capacity
8. The Agile Approach
9. Finding our Target Condition Feedback Expectations Customers Service/Product
10. Feedback Expectations Customers Service/Product • What do we deliver? Finding our Target Condition
11. Feedback Expectations Customers Service/Product • What do we deliver? • For whom do we produce it? Who defines our work? Finding our Target Condition
12. Feedback Expectations Customers Service/Product • What do we deliver? • For whom do we produce it? Who defines our work? • What do they expect of us? Finding our Target Condition
13. Feedback Expectations Customers Service/Product • What do we deliver? • For whom do we produce it? Who defines our work? • What do they expect of us? • Did we meet those expectations? Finding our Target Condition
14. Implementing Feedback Loops
15. Metrics we wanted to see ‣ Signup/auth flow abandonment ‣ Facebook-connected users – percentage of total users, trend over time ‣ Percent of users that sign up through Facebook per hour/day/ week ‣ Facebook-related errors
16. Metrics we wanted to see ‣ Daily Active Users by Partner/Feature ‣ Registrations, subscription rate, and referrals by Partner ‣ Web API usage by Partner ‣ Squad-focused Twitter feed ("uber + spotify," etc.) – what’s being complained about that neither the partner or we may not see? ‣ Outstanding JIRA issues ‣ Request count by internal requesting service/team
17. New Processes ‣ Evaluate metrics during retrospectives. ‣ Set metric goals to new integration projects. ‣ Soliciting feedback from external partners.
18. The Big Picture
19. Creating a new metric ‣ How do metrics map to business goals? ‣ How do you prioritize different goals you want to drive? ‣ How can we create dashboards that are actually actionable?
20. Representing metrics ‣ How do we correctly measure what we care about? ‣ What’s best: a gauge, counter, meter, histogram, or a timer? ‣ What’s a better representation for a metric, the mean, 95% 99%?
21. Consuming metrics ‣ How often do you check in on metrics? ‣ How do you make dashboards more visible? ‣ Considerations for the setup?
22. Iterating over current metrics ‣ Why did we not reach our goal? ‣ What action are we going to take? ‣ What about unknowns?
23. The Big Picture
24. TL;DR: If it moves, you can track it.
 But should you?
25. TL;DR: If it moves, you can track it.
 Probably, but only thoughtfully.
26. Thank you. rogue.ly/metrics
 @roguelynn image credits: rogue.ly/mmd-images