Save 37% off PRO during our Black Friday Sale! »

The Great PagerDuty DevOps Heist of 2016

The Great PagerDuty DevOps Heist of 2016

This is a talk I gave at DevOpsDays Stockholm 2017
https://www.devopsdays.org/events/2017-stockholm/program/arup-chakrabarti/

Ebe1d126c7c859171156efb4c08db14f?s=128

Arup Chakrabarti

May 09, 2017
Tweet

Transcript

  1. @arupchak The Great PagerDuty DevOps Heist
 of 2016 Arup Chakrabarti

    Director of Engineering DEVOPSDAYS STOCKHOLM
  2. @arupchak Agenda

  3. @arupchak Who is this guy? • Academic Research • Amazon

    • Netflix • PagerDuty
  4. @arupchak Disclaimer: I am not a smrt person (I work

    with smart people)
  5. @arupchak Disclaimer: This will not work for you

  6. @arupchak What is PagerDuty?

  7. @arupchak Unhappy Customers Magic Resolvers Incident Management

  8. @arupchak January 2016

  9. @arupchak Org Chart VP of Engineering Operations Servers Databases LB’s

    Reliability Backend services System reliability Product Customer facing Frontend services
  10. @arupchak Problems • Centralized Operations Responsibility • Centralized Reliability Responsibility

    • Centralized Product Responsibility
  11. @arupchak “The Heist”

  12. @arupchak “The Heist” • What we were selling to teams

    • Business Success depends on innovation • Innovation depends on rate of change • We will increase the rate of change by having engineers own more
  13. @arupchak March 2016

  14. @arupchak Org Chart VP of Engineering Infrastructure Products SRE Platform

    Security Product Team Alpha Servers Databases Operations Product Team Beta Servers Databases Operations
  15. @arupchak More Problems • People working with new people •

    People working on things they were not hired for • Changed responsibilities for everyone
  16. @arupchak What Changed?

  17. @arupchak The Job Changed • For previously Dev focused people

    • They now owned the full vertical stack • Code It. Ship It. Own It. • The previously Ops focused people • They now had to empower others to do their job • Make the right thing the easy thing
  18. @arupchak “I need to do what now?” -Every Engineer

  19. @arupchak Self Service Tooling

  20. @arupchak What we built • Infrastructure Tools • Self Service

    Server Provisioning • Self Service Metrics and Telemetry • Self Service Deployments • Self Service Common Infrastructure Services • Documentation • Where we could not automate easily
  21. @arupchak “Ok, I think I can handle that” -Most Engineers

  22. @arupchak People

  23. @arupchak “This sucks, I am out of here” -Some Engineers

  24. @arupchak The Hard Part of Leadership • Some changes are

    not for everyone • Some people who thrived in the old ways, will fail in the new ways • They are not trying to be jerks • Expect 10% attrition or managed exits
  25. @arupchak May 2017

  26. @arupchak “Despite Arup’s efforts, PagerDuty is still in business”

  27. @arupchak What progress looks like (Did it work?)

  28. @arupchak Github Metrics

  29. @arupchak Chef Repo Metrics

  30. @arupchak PagerDuty Metrics

  31. @arupchak Business Metrics

  32. @arupchak Anecdotes • Intern replaced our Kafka cluster • Engineers

    practicing MySQL recovery • Engineers building out test coverage metrics
  33. @arupchak “The Heist”

  34. @arupchak “The Heist” • What we were selling to teams

    • Business Success depends on innovation • Innovation depends on rate of change • We will increase the rate of change by having engineers own more
  35. @arupchak “The Heist” • What did it cost the teams?

    • All Engineers go on-call • Code it. Ship it. Own it • People leaving
  36. @arupchak Did we get it completely right?

  37. @arupchak Of Course Not

  38. @arupchak Thank You! WE ARE HIRING! JOBS.PAGERDUTY.COM Arup Chakrabarti Director

    of Engineering