The Great PagerDuty DevOps Heist of 2016

This is a talk I gave at DevOpsDays Stockholm 2017
https://www.devopsdays.org/events/2017-stockholm/program/arup-chakrabarti/

Transcript

1. @arupchak The Great PagerDuty DevOps Heist
 of 2016 Arup Chakrabarti Director of Engineering DEVOPSDAYS STOCKHOLM
2. @arupchak Agenda
3. @arupchak Who is this guy? • Academic Research • Amazon • Netflix • PagerDuty
4. @arupchak Disclaimer: I am not a smrt person (I work with smart people)
5. @arupchak Disclaimer: This will not work for you
6. @arupchak What is PagerDuty?
7. @arupchak Unhappy Customers Magic Resolvers Incident Management
8. @arupchak January 2016
9. @arupchak Org Chart VP of Engineering Operations Servers Databases LB’s Reliability Backend services System reliability Product Customer facing Frontend services
10. @arupchak Problems • Centralized Operations Responsibility • Centralized Reliability Responsibility • Centralized Product Responsibility
11. @arupchak “The Heist”
12. @arupchak “The Heist” • What we were selling to teams • Business Success depends on innovation • Innovation depends on rate of change • We will increase the rate of change by having engineers own more
13. @arupchak March 2016
14. @arupchak Org Chart VP of Engineering Infrastructure Products SRE Platform Security Product Team Alpha Servers Databases Operations Product Team Beta Servers Databases Operations
15. @arupchak More Problems • People working with new people • People working on things they were not hired for • Changed responsibilities for everyone
16. @arupchak What Changed?
17. @arupchak The Job Changed • For previously Dev focused people • They now owned the full vertical stack • Code It. Ship It. Own It. • The previously Ops focused people • They now had to empower others to do their job • Make the right thing the easy thing
18. @arupchak “I need to do what now?” -Every Engineer
19. @arupchak Self Service Tooling
20. @arupchak What we built • Infrastructure Tools • Self Service Server Provisioning • Self Service Metrics and Telemetry • Self Service Deployments • Self Service Common Infrastructure Services • Documentation • Where we could not automate easily
21. @arupchak “Ok, I think I can handle that” -Most Engineers
22. @arupchak People
23. @arupchak “This sucks, I am out of here” -Some Engineers
24. @arupchak The Hard Part of Leadership • Some changes are not for everyone • Some people who thrived in the old ways, will fail in the new ways • They are not trying to be jerks • Expect 10% attrition or managed exits
25. @arupchak May 2017
26. @arupchak “Despite Arup’s efforts, PagerDuty is still in business”
27. @arupchak What progress looks like (Did it work?)
28. @arupchak Github Metrics
29. @arupchak Chef Repo Metrics
30. @arupchak PagerDuty Metrics
31. @arupchak Business Metrics
32. @arupchak Anecdotes • Intern replaced our Kafka cluster • Engineers practicing MySQL recovery • Engineers building out test coverage metrics
33. @arupchak “The Heist”
34. @arupchak “The Heist” • What we were selling to teams • Business Success depends on innovation • Innovation depends on rate of change • We will increase the rate of change by having engineers own more
35. @arupchak “The Heist” • What did it cost the teams? • All Engineers go on-call • Code it. Ship it. Own it • People leaving
36. @arupchak Did we get it completely right?
37. @arupchak Of Course Not
38. @arupchak Thank You! WE ARE HIRING! JOBS.PAGERDUTY.COM Arup Chakrabarti Director of Engineering