$30 off During Our Annual Pro Sale. View Details »

"Why would anyone do out-of-hours support for free?"

Sarah Wells
September 19, 2016

"Why would anyone do out-of-hours support for free?"

To successfully move to DevOps, you will need to change your company's culture in a lot of ways. If you have people in very distinct operations and development teams, how do you convince them about the benefits of closer collaboration and blurring of lines?

Why would operations let developers have production access? If it resulted in better monitored, better documented, more stable and resilient systems, maybe they'll accept the perceived extra risk.

Why would a developer accept being woken up at 2 am for no more money? If it means having the power to make more decisions about the tools, processes and software to use, maybe they'll be fine with that.

Over the last few years at the Financial Times, we've gone through this culture change, and I'm happy to share some of the problems we've faced and the solutions we've tried.

Sarah Wells

September 19, 2016

More Decks by Sarah Wells

Other Decks in Programming


  1. "Why would anyone do out- of-hours support for free?" Sarah

    Wells Principal Engineer, Financial Times @sarahjwells
  2. @sarahjwells Hello

  3. @sarahjwells Why this title?

  4. What I learned about devops at the FT

  5. @sarahjwells 1. Why devops? 2. What we did 3. The

    hard stuff
  6. @sarahjwells 1. Why devops? 2. What we did 3. The

    hard stuff
  7. @sarahjwells First: the Financial Times

  8. None
  9. @sarahjwells “… one of the world's leading business news and

    information organisations…”
  10. @sarahjwells We need to be able to move fast

  11. @sarahjwells Why did we think devops would help?

  12. @sarahjwells https://puppet.com/resources/white-paper/2016-state-devops-report/

  13. @sarahjwells But nothing comes for free

  14. @sarahjwells Doing devops properly means changing your culture

  15. @sarahjwells What motivated us to change?

  16. @sarahjwells People were frustrated

  17. @sarahjwells Slow

  18. @sarahjwells It took months to get a production server

  19. @sarahjwells It took days to get a release ready

  20. None
  21. @sarahjwells It took hours to do that release

  22. @sarahjwells And we did those releases once a month

  23. @sarahjwells Processes were manual and error prone

  24. @sarahjwells All our servers were snowflake servers

  25. @sarahjwells Release instructions were in a spreadsheet

  26. None
  27. @sarahjwells Releasing was a box ticking exercise

  28. @sarahjwells Preconditions for change

  29. @sarahjwells A change in how the business thought about technology

  30. None
  31. @sarahjwells Big, complex new projects

  32. @sarahjwells An in-house example of doing things differently

  33. @sarahjwells 1. Why devops? 2. What we did 3. The

    hard stuff
  34. @sarahjwells Infrastructure as code

  35. @sarahjwells Developers could provision VMs

  36. @sarahjwells Built-in support for the things that mattered

  37. None
  38. None
  39. @sarahjwells Collaboration

  40. @sarahjwells

  41. @sarahjwells Tech ops joined teams

  42. @sarahjwells Groups of people with a common interest

  43. @sarahjwells “You build it, you run it”

  44. @sarahjwells If you aren’t doing this, you aren’t doing devops

  45. @sarahjwells This made developers focus on building for operability

  46. @sarahjwells Support via an “engineering checklist”

  47. None
  48. @sarahjwells Started by handling operations “in hours”

  49. @sarahjwells “Ops cops” and a dedicated kanban lane

  50. @sarahjwells Providing quick ways to find out about and investigate

  51. None
  52. None
  53. @sarahjwells Need to share knowledge widely

  54. @sarahjwells Programme ‘curriculums’

  55. None
  56. None
  57. @sarahjwells 1. Why devops? 2. What we did 3. The

    hard stuff
  58. @sarahjwells The most difficult thing to do is change your

  59. @sarahjwells Persuading operations to trust developers

  60. @sarahjwells I could provision a VM but not do the

    first deploy
  61. @sarahjwells Development teams needed to prove we cared about operability

  62. @sarahjwells It helped having second line support on the teams

  63. @sarahjwells Empowering teams, not just talking about it

  64. @sarahjwells Who makes the technology decisions?

  65. @sarahjwells Empowerment also means the freedom NOT to use internal

  66. @sarahjwells –http://matt.chadburn.co.uk/notes/teams-as-services.html “the choice to decide who is going to

    provide a service is typically around ease of use (for the team), readiness, and self-sufficiency, rather than the being too concerned about the politics of vertical integration across the company”
  67. @sarahjwells Tools and standards are better than platforms

  68. None
  69. @sarahjwells Out-of-hours support

  70. @sarahjwells Operations teams worry about their roles changing

  71. @sarahjwells Developers worry about having to do a second shift

  72. @sarahjwells There are lots of reasons people may not want

    to be on call
  73. @sarahjwells So what have we settled on?

  74. @sarahjwells We still have first and second line support on

    duty or on call
  75. None
  76. @sarahjwells It’s not one size fits all

  77. @sarahjwells Small teams vs large teams

  78. @sarahjwells Don’t be afraid to try things out

  79. @sarahjwells Don’t treat every decision like it’s irreversible http://uk.businessinsider.com/jeff-bezos-on-type-1-and-type-2-decisions-2016-4

  80. @sarahjwells So why *would* someone do out-of-hours support for free?

  81. @sarahjwells What difference has devops made at the FT?

  82. @sarahjwells A new service can be put into production in

    minutes not months
  83. @sarahjwells Zero down time deployments are normal

  84. @sarahjwells Minutes from test sign off to live in production

  85. @sarahjwells My team have released to production over 1400 times

    this year
  86. None
  87. None
  88. None
  89. @sarahjwells Our Google AMP integration was built in weeks

  90. @sarahjwells 1. Why devops? 2. What we did 3. The

    hard stuff
  91. @sarahjwells Thank you!