"Why would anyone do out-of-hours support for free?"

Slide 1

Slide 1 text

"Why would anyone do out- of-hours support for free?" Sarah Wells Principal Engineer, Financial Times @sarahjwells

Slide 2

Slide 2 text

@sarahjwells Hello

Slide 3

Slide 3 text

@sarahjwells Why this title?

Slide 4

Slide 4 text

What I learned about devops at the FT

Slide 5

Slide 5 text

@sarahjwells 1. Why devops? 2. What we did 3. The hard stuﬀ

Slide 6

Slide 6 text

@sarahjwells 1. Why devops? 2. What we did 3. The hard stuﬀ

Slide 7

Slide 7 text

@sarahjwells First: the Financial Times

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

@sarahjwells “… one of the world's leading business news and information organisations…”

Slide 10

Slide 10 text

@sarahjwells We need to be able to move fast

Slide 11

Slide 11 text

@sarahjwells Why did we think devops would help?

Slide 12

Slide 12 text

@sarahjwells https://puppet.com/resources/white-paper/2016-state-devops-report/

Slide 13

Slide 13 text

@sarahjwells But nothing comes for free

Slide 14

Slide 14 text

@sarahjwells Doing devops properly means changing your culture

Slide 15

Slide 15 text

@sarahjwells What motivated us to change?

Slide 16

Slide 16 text

@sarahjwells People were frustrated

Slide 17

Slide 17 text

@sarahjwells Slow

Slide 18

Slide 18 text

@sarahjwells It took months to get a production server

Slide 19

Slide 19 text

@sarahjwells It took days to get a release ready

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

@sarahjwells It took hours to do that release

Slide 22

Slide 22 text

@sarahjwells And we did those releases once a month

Slide 23

Slide 23 text

@sarahjwells Processes were manual and error prone

Slide 24

Slide 24 text

@sarahjwells All our servers were snowﬂake servers

Slide 25

Slide 25 text

@sarahjwells Release instructions were in a spreadsheet

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

@sarahjwells Releasing was a box ticking exercise

Slide 28

Slide 28 text

@sarahjwells Preconditions for change

Slide 29

Slide 29 text

@sarahjwells A change in how the business thought about technology

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

@sarahjwells Big, complex new projects

Slide 32

Slide 32 text

@sarahjwells An in-house example of doing things diﬀerently

Slide 33

Slide 33 text

@sarahjwells 1. Why devops? 2. What we did 3. The hard stuﬀ

Slide 34

Slide 34 text

@sarahjwells Infrastructure as code

Slide 35

Slide 35 text

@sarahjwells Developers could provision VMs

Slide 36

Slide 36 text

@sarahjwells Built-in support for the things that mattered

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

@sarahjwells Collaboration

Slide 40

Slide 40 text

@sarahjwells

Slide 41

Slide 41 text

@sarahjwells Tech ops joined teams

Slide 42

Slide 42 text

@sarahjwells Groups of people with a common interest

Slide 43

Slide 43 text

@sarahjwells “You build it, you run it”

Slide 44

Slide 44 text

@sarahjwells If you aren’t doing this, you aren’t doing devops

Slide 45

Slide 45 text

@sarahjwells This made developers focus on building for operability

Slide 46

Slide 46 text

@sarahjwells Support via an “engineering checklist”

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

@sarahjwells Started by handling operations “in hours”

Slide 49

Slide 49 text

@sarahjwells “Ops cops” and a dedicated kanban lane

Slide 50

Slide 50 text

@sarahjwells Providing quick ways to ﬁnd out about and investigate issues

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

@sarahjwells Need to share knowledge widely

Slide 54

Slide 54 text

@sarahjwells Programme ‘curriculums’

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

@sarahjwells 1. Why devops? 2. What we did 3. The hard stuﬀ

Slide 58

Slide 58 text

@sarahjwells The most difficult thing to do is change your culture

Slide 59

Slide 59 text

@sarahjwells Persuading operations to trust developers

Slide 60

Slide 60 text

@sarahjwells I could provision a VM but not do the ﬁrst deploy

Slide 61

Slide 61 text

@sarahjwells Development teams needed to prove we cared about operability

Slide 62

Slide 62 text

@sarahjwells It helped having second line support on the teams

Slide 63

Slide 63 text

@sarahjwells Empowering teams, not just talking about it

Slide 64

Slide 64 text

@sarahjwells Who makes the technology decisions?

Slide 65

Slide 65 text

@sarahjwells Empowerment also means the freedom NOT to use internal services

Slide 66

Slide 66 text

@sarahjwells –http://matt.chadburn.co.uk/notes/teams-as-services.html “the choice to decide who is going to provide a service is typically around ease of use (for the team), readiness, and self-sufficiency, rather than the being too concerned about the politics of vertical integration across the company”

Slide 67

Slide 67 text

@sarahjwells Tools and standards are better than platforms

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

@sarahjwells Out-of-hours support

Slide 70

Slide 70 text

@sarahjwells Operations teams worry about their roles changing

Slide 71

Slide 71 text

@sarahjwells Developers worry about having to do a second shift

Slide 72

Slide 72 text

@sarahjwells There are lots of reasons people may not want to be on call

Slide 73

Slide 73 text

@sarahjwells So what have we settled on?

Slide 74

Slide 74 text

@sarahjwells We still have ﬁrst and second line support on duty or on call

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

@sarahjwells It’s not one size ﬁts all

Slide 77

Slide 77 text

@sarahjwells Small teams vs large teams

Slide 78

Slide 78 text

@sarahjwells Don’t be afraid to try things out

Slide 79

Slide 79 text

@sarahjwells Don’t treat every decision like it’s irreversible http://uk.businessinsider.com/jeﬀ-bezos-on-type-1-and-type-2-decisions-2016-4