Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Metrics-Driven Development: See the forest for the trees

Metrics-Driven Development: See the forest for the trees

UPDATED SLIDES: October 2015 - GitHub Universe

Talk originally given at Open Data Science in Boston, May 2015 and EuroPython in July 2015.

Full blog post: http://rogue.ly/metrics

Lynn Root

May 30, 2015
Tweet

More Decks by Lynn Root

Other Decks in Technology

Transcript

  1. Metrics-Driven
    Development
    Lynn Root

    @roguelynn

    View full-size slide

  2. $ whoami
    • Backend Engineer @ Spotify, based in San Fransisco
    • Board Member of Python Software Foundation
    • Founder of PyLadies of San Francisco, one of the lead organizers
    of PyLadies globally

    View full-size slide

  3. • About Spotify
    • Our agile approach to & implementation of metrics
    • Stepping back for the bigger picture
    See the forest for the trees.

    View full-size slide

  4. TL;DR: If it moves,
    you can track it.

    But should you?

    View full-size slide

  5. About Spotify

    View full-size slide

  6. About Spotify.
    • Available in 58 countries
    • Over 20 million paying subscribers
    • Over 75 million monthly active users
    • Over 30 million songs; 20,000
    added each day
    • Paid over $3 billion to rights holders
    to date
    • Main engineering offices located in
    Stockholm & New York 


    View full-size slide

  7. About Spotify.
    • 30 TB of user & service-related log
    data per day
    • 400 TB of data generated in Hadoop
    each day
    • 1600+ node Hadoop cluster
    • 17,000 Hadoop jobs per day
    • 60 PBs of storage capacity

    View full-size slide

  8. The Agile
    Approach

    View full-size slide

  9. Finding our Target Condition
    Feedback
    Expectations
    Customers
    Service/Product

    View full-size slide

  10. Feedback
    Expectations
    Customers
    Service/Product
    • What do we deliver?
    Finding our Target Condition

    View full-size slide

  11. Feedback
    Expectations
    Customers
    Service/Product
    • What do we deliver?
    • For whom do we produce it? Who defines our work?
    Finding our Target Condition

    View full-size slide

  12. Feedback
    Expectations
    Customers
    Service/Product
    • What do we deliver?
    • For whom do we produce it? Who defines our work?
    • What do they expect of us?
    Finding our Target Condition

    View full-size slide

  13. Feedback
    Expectations
    Customers
    Service/Product
    • What do we deliver?
    • For whom do we produce it? Who defines our work?
    • What do they expect of us?
    • Did we meet those expectations?
    Finding our Target Condition

    View full-size slide

  14. Implementing
    Feedback Loops

    View full-size slide

  15. Metrics we wanted to see
    ‣ Signup/auth flow abandonment
    ‣ Facebook-connected users – percentage of total users, trend
    over time
    ‣ Percent of users that sign up through Facebook per hour/day/
    week
    ‣ Facebook-related errors

    View full-size slide

  16. Metrics we wanted to see
    ‣ Daily Active Users by Partner/Feature
    ‣ Registrations, subscription rate, and referrals by Partner
    ‣ Web API usage by Partner
    ‣ Squad-focused Twitter feed ("uber + spotify," etc.) – what’s being
    complained about that neither the partner or we may not see?
    ‣ Outstanding JIRA issues
    ‣ Request count by internal requesting service/team

    View full-size slide

  17. New Processes
    ‣ Evaluate metrics during retrospectives.
    ‣ Set metric goals to new integration projects.
    ‣ Soliciting feedback from external partners.

    View full-size slide

  18. The Big Picture

    View full-size slide

  19. Creating a new metric
    ‣ How do metrics map to business goals?
    ‣ How do you prioritize different goals you want to
    drive?
    ‣ How can we create dashboards that are actually
    actionable?

    View full-size slide

  20. Representing metrics
    ‣ How do we correctly measure what we care about?
    ‣ What’s best: a gauge, counter, meter, histogram, or a
    timer?
    ‣ What’s a better representation for a metric, the mean,
    95% 99%?

    View full-size slide

  21. Consuming metrics
    ‣ How often do you check in on metrics?
    ‣ How do you make dashboards more visible?
    ‣ Considerations for the setup?

    View full-size slide

  22. Iterating over current metrics
    ‣ Why did we not reach our goal?
    ‣ What action are we going to take?
    ‣ What about unknowns?

    View full-size slide

  23. The Big Picture

    View full-size slide

  24. TL;DR: If it moves,
    you can track it.

    But should you?

    View full-size slide

  25. TL;DR: If it moves,
    you can track it.

    Probably, but only
    thoughtfully.

    View full-size slide

  26. Thank you.
    rogue.ly/metrics

    @roguelynn
    image credits: rogue.ly/mmd-images

    View full-size slide