Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Instrumenting The Rest Of The Company: Hunting For Metrics

Instrumenting The Rest Of The Company: Hunting For Metrics

Presented at Monitorama 2017, video at: https://youtu.be/wnjCNBfH3kg?t=3h3m35s


Eric Sigler

May 23, 2017


  1. @esigler Instrumenting The Rest Of The Company: Hunting For Useful

    Metrics Eric Sigler, Head of DevOps, PagerDuty
  2. @esigler Alternatively: ”Lies, Damn Lies, and Hacky Scripts"

  3. @esigler

  4. @esigler Engineer Eng Engineer Eng? Manager Mgr Manager

  5. @esigler (No stock photos harmed in the making of this

  6. @esigler "We have problem $foo, so we're going to do

  7. @esigler "What data did you use to understand $foo? And

    how will we know if $bar improved anything?”
  8. @esigler “We can’t really measure
 either $foo and/or $bar.”

  9. @esigler “Without data, you're just another person with an opinion.”

    - W. Edwards Deming
  10. @esigler

  11. @esigler

  12. @esigler

  13. @esigler (Turns out other managers do this too.)

  14. @esigler

  15. @esigler "We have a problem with people not knowing what

    the chatbot does, so we're going to write better documentation."
  16. @esigler

  17. @esigler ?

  18. @esigler

  19. @esigler “If only there was some way we could track

    events, and show them over time.”
  20. @esigler

  21. @esigler

  22. @esigler

  23. @esigler Outcome: Writing a smarter help function in the chat

    bot. (And simplifying some commands).
  24. @esigler

  25. @esigler Takeaway: Reuse existing tools when it makes sense.

  26. @esigler

  27. @esigler "We have slow tests in CI, so we're going

    to complain a lot about it.”
  28. @esigler “Define slow.”

  29. @esigler Local != CI

  30. @esigler

  31. @esigler

  32. @esigler

  33. @esigler

  34. @esigler “Tests take forever to start.”

  35. @esigler

  36. @esigler ?

  37. @esigler

  38. @esigler Outcome: More workers. (And, knowing how many to budget

  39. @esigler Takeaway: Look for ways to reverse engineer existing metrics.

  40. @esigler

  41. @esigler "We have to ship code faster, so we're going

    to reorganize."
  42. @esigler

  43. @esigler

  44. @esigler But it doesn’t show where the bottlenecks are.

  45. @esigler Pipe GitHub metrics into

  46. @esigler

  47. @esigler

  48. @esigler

  49. @esigler Then start making changes.

  50. @esigler

  51. @esigler

  52. @esigler Outcome: Productivity success! (With massive organizational change to enable

  53. @esigler Takeaway: Look for proxy metrics

  54. @esigler Potpourri: Data collection (chat, email, calendars) Cross-validation of metrics

    (“Sniff test”) Cognitive biases around metrics Plotting against organization events
  55. @esigler Takeaways: Useful metrics are everywhere You aren’t alone in

    digging for metrics Existing tools can be repurposed Look to reverse engineer your way to a metric Look for proxy metrics (but choose wisely)
  56. @esigler Thank you!

  57. @esigler Image credits:
 https://commons.wikimedia.org/wiki/File:Staff_meeting.jpg https://blogs-images.forbes.com/kellyallan/files/2015/06/Deming-in-Tuxedo-DEM-1078-Dr.-Deming2-1940x1130.jpg (Wherever I grabbed that

    screenshot from Pulp Fiction, my apologies I am a terrible person for not capturing the URL)