Instrumenting The Rest Of The Company: Hunting For Metrics

Transcript

1. 1.

   @esigler Instrumenting The Rest Of The Company: Hunting For Useful

   Metrics Eric Sigler, Head of DevOps, PagerDuty
2. 2.

   @esigler Alternatively: ”Lies, Damn Lies, and Hacky Scripts"

3. 3.

   @esigler

4. 4.

   @esigler Engineer Eng Engineer Eng? Manager Mgr Manager

5. 5.

   @esigler (No stock photos harmed in the making of this

   talk.)
6. 6.

   @esigler "We have problem $foo, so we're going to do

   $bar."
7. 7.

   @esigler "What data did you use to understand $foo? And

   how will we know if $bar improved anything?”
8. 8.

   @esigler “We can’t really measure
 either $foo and/or $bar.”

9. 9.

   @esigler “Without data, you're just another person with an opinion.”

   - W. Edwards Deming
10. 10.

    @esigler

11. 11.

    @esigler

12. 12.

    @esigler

13. 13.

    @esigler (Turns out other managers do this too.)

14. 14.

    @esigler

15. 15.

    @esigler "We have a problem with people not knowing what

    the chatbot does, so we're going to write better documentation."
16. 16.

    @esigler

17. 17.

    @esigler ?

18. 18.

    @esigler

19. 19.

    @esigler “If only there was some way we could track

    events, and show them over time.”
20. 20.

    @esigler

21. 21.

    @esigler

22. 22.

    @esigler

23. 23.

    @esigler Outcome: Writing a smarter help function in the chat

    bot. (And simplifying some commands).
24. 24.

    @esigler

25. 25.

    @esigler Takeaway: Reuse existing tools when it makes sense.

26. 26.

    @esigler

27. 27.

    @esigler "We have slow tests in CI, so we're going

    to complain a lot about it.”
28. 28.

    @esigler “Define slow.”

29. 29.

    @esigler Local != CI

30. 30.

    @esigler

31. 31.

    @esigler

32. 32.

    @esigler

33. 33.

    @esigler

34. 34.

    @esigler “Tests take forever to start.”

35. 35.

    @esigler

36. 36.

    @esigler ?

37. 37.

    @esigler

38. 38.

    @esigler Outcome: More workers. (And, knowing how many to budget

    for.)
39. 39.

    @esigler Takeaway: Look for ways to reverse engineer existing metrics.

40. 40.

    @esigler

41. 41.

    @esigler "We have to ship code faster, so we're going

    to reorganize."
42. 42.

    @esigler

43. 43.

    @esigler

44. 44.

    @esigler But it doesn’t show where the bottlenecks are.

45. 45.

    @esigler Pipe GitHub metrics into
 &

46. 46.

    @esigler

47. 47.

    @esigler

48. 48.

    @esigler

49. 49.

    @esigler Then start making changes.

50. 50.

    @esigler

51. 51.

    @esigler

52. 52.

    @esigler Outcome: Productivity success! (With massive organizational change to enable

    it.)
53. 53.

    @esigler Takeaway: Look for proxy metrics

54. 54.

    @esigler Potpourri: Data collection (chat, email, calendars) Cross-validation of metrics

    (“Sniff test”) Cognitive biases around metrics Plotting against organization events
55. 55.

    @esigler Takeaways: Useful metrics are everywhere You aren’t alone in

    digging for metrics Existing tools can be repurposed Look to reverse engineer your way to a metric Look for proxy metrics (but choose wisely)
56. 56.

    @esigler Thank you!

57. 57.

    @esigler Image credits:
 
 https://commons.wikimedia.org/wiki/File:Staff_meeting.jpg https://blogs-images.forbes.com/kellyallan/files/2015/06/Deming-in-Tuxedo-DEM-1078-Dr.-Deming2-1940x1130.jpg (Wherever I grabbed that

    screenshot from Pulp Fiction, my apologies I am a terrible person for not capturing the URL)