"Why would anyone do out-of-hours support for free?"

To successfully move to DevOps, you will need to change your company's culture in a lot of ways. If you have people in very distinct operations and development teams, how do you convince them about the benefits of closer collaboration and blurring of lines?

Why would operations let developers have production access? If it resulted in better monitored, better documented, more stable and resilient systems, maybe they'll accept the perceived extra risk.

Why would a developer accept being woken up at 2 am for no more money? If it means having the power to make more decisions about the tools, processes and software to use, maybe they'll be fine with that.

Over the last few years at the Financial Times, we've gone through this culture change, and I'm happy to share some of the problems we've faced and the solutions we've tried.

Transcript

1. "Why would anyone do out-
   of-hours support for free?"
   Sarah Wells
   Principal Engineer, Financial Times
   @sarahjwells
   

   View Slide

2. @sarahjwells
   Hello
   

   View Slide

3. @sarahjwells
   Why this title?
   

   View Slide

4. What I learned about
   devops at the FT
   

   View Slide

5. @sarahjwells
   1. Why devops?
   2. What we did
   3. The hard stuff
   

   View Slide

6. @sarahjwells
   1. Why devops?
   2. What we did
   3. The hard stuff
   

   View Slide

7. @sarahjwells
   First: the Financial Times
   

   View Slide

8. View Slide

9. @sarahjwells
   “… one of the world's leading
   business news and information
   organisations…”
   

   View Slide

10. @sarahjwells
    We need to be able to move fast
    

    View Slide

11. @sarahjwells
    Why did we think devops would help?
    

    View Slide

12. @sarahjwells https://puppet.com/resources/white-paper/2016-state-devops-report/
    

    View Slide

13. @sarahjwells
    But nothing comes for free
    

    View Slide

14. @sarahjwells
    Doing devops properly means
    changing your culture
    

    View Slide

15. @sarahjwells
    What motivated us to change?
    

    View Slide

16. @sarahjwells
    People were frustrated
    

    View Slide

17. @sarahjwells
    Slow
    

    View Slide

18. @sarahjwells
    It took months to get a production server
    

    View Slide

19. @sarahjwells
    It took days to get a release ready
    

    View Slide

20. View Slide

21. @sarahjwells
    It took hours to do that release
    

    View Slide

22. @sarahjwells
    And we did those releases once a month
    

    View Slide

23. @sarahjwells
    Processes were manual and error prone
    

    View Slide

24. @sarahjwells
    All our servers were snowflake servers
    

    View Slide

25. @sarahjwells
    Release instructions were in a spreadsheet
    

    View Slide

26. View Slide

27. @sarahjwells
    Releasing was a box ticking exercise
    

    View Slide

28. @sarahjwells
    Preconditions for change
    

    View Slide

29. @sarahjwells
    A change in how the business thought about
    technology
    

    View Slide

30. View Slide

31. @sarahjwells
    Big, complex new projects
    

    View Slide

32. @sarahjwells
    An in-house example of doing things differently
    

    View Slide

33. @sarahjwells
    1. Why devops?
    2. What we did
    3. The hard stuff
    

    View Slide

34. @sarahjwells
    Infrastructure as code
    

    View Slide

35. @sarahjwells
    Developers could provision VMs
    

    View Slide

36. @sarahjwells
    Built-in support for the things that mattered
    

    View Slide

37. View Slide

38. View Slide

39. @sarahjwells
    Collaboration
    

    View Slide

40. @sarahjwells
    

    View Slide

41. @sarahjwells
    Tech ops joined teams
    

    View Slide

42. @sarahjwells
    Groups of people with a common interest
    

    View Slide

43. @sarahjwells
    “You build it, you run it”
    

    View Slide

44. @sarahjwells
    If you aren’t doing this, you aren’t doing devops
    

    View Slide

45. @sarahjwells
    This made developers focus on building for
    operability
    

    View Slide

46. @sarahjwells
    Support via an “engineering checklist”
    

    View Slide

47. View Slide

48. @sarahjwells
    Started by handling operations “in hours”
    

    View Slide

49. @sarahjwells
    “Ops cops” and a dedicated kanban lane
    

    View Slide

50. @sarahjwells
    Providing quick ways to find out about and
    investigate issues
    

    View Slide

51. View Slide

52. View Slide

53. @sarahjwells
    Need to share knowledge widely
    

    View Slide

54. @sarahjwells
    Programme ‘curriculums’
    

    View Slide

55. View Slide

56. View Slide

57. @sarahjwells
    1. Why devops?
    2. What we did
    3. The hard stuff
    

    View Slide

58. @sarahjwells
    The most difficult thing to do is
    change your culture
    

    View Slide

59. @sarahjwells
    Persuading operations to trust developers
    

    View Slide

60. @sarahjwells
    I could provision a VM but not do the first deploy
    

    View Slide

61. @sarahjwells
    Development teams needed to prove we cared
    about operability
    

    View Slide

62. @sarahjwells
    It helped having second line support on the teams
    

    View Slide

63. @sarahjwells
    Empowering teams, not just talking about it
    

    View Slide

64. @sarahjwells
    Who makes the technology decisions?
    

    View Slide

65. @sarahjwells
    Empowerment also means the freedom NOT to use
    internal services
    

    View Slide

66. @sarahjwells
    –http://matt.chadburn.co.uk/notes/teams-as-services.html
    “the choice to decide who is going to provide a
    service is typically around ease of use (for the team),
    readiness, and self-sufficiency, rather than the being
    too concerned about the politics of vertical
    integration across the company”
    

    View Slide

67. @sarahjwells
    Tools and standards are better than platforms
    

    View Slide

68. View Slide

69. @sarahjwells
    Out-of-hours support
    

    View Slide

70. @sarahjwells
    Operations teams worry about their roles changing
    

    View Slide

71. @sarahjwells
    Developers worry about having to do a second shift
    

    View Slide

72. @sarahjwells
    There are lots of reasons people may not want to be
    on call
    

    View Slide

73. @sarahjwells
    So what have we settled on?
    

    View Slide

74. @sarahjwells
    We still have first and second line support on duty or
    on call
    

    View Slide

75. View Slide

76. @sarahjwells
    It’s not one size fits all
    

    View Slide

77. @sarahjwells
    Small teams vs large teams
    

    View Slide

78. @sarahjwells
    Don’t be afraid to try things out
    

    View Slide

79. @sarahjwells
    Don’t treat every decision like
    it’s irreversible
    http://uk.businessinsider.com/jeff-bezos-on-type-1-and-type-2-decisions-2016-4
    

    View Slide

80. @sarahjwells
    So why *would* someone do out-of-hours support
    for free?
    

    View Slide

81. @sarahjwells
    What difference has devops made at
    the FT?
    

    View Slide

82. @sarahjwells
    A new service can be put into production in minutes
    not months
    

    View Slide

83. @sarahjwells
    Zero down time deployments are normal
    

    View Slide

84. @sarahjwells
    Minutes from test sign off to live in production
    

    View Slide

85. @sarahjwells
    My team have released to production over 1400
    times this year
    

    View Slide

86. View Slide

87. View Slide

88. View Slide

89. @sarahjwells
    Our Google AMP integration was built in weeks
    

    View Slide

90. @sarahjwells
    1. Why devops?
    2. What we did
    3. The hard stuff
    

    View Slide

91. @sarahjwells
    Thank you!
    

    View Slide