DevOps in the hell of a thousand different platforms Sam Eaton Sunday, 17 March 2013

Who am I? • Sam Eaton • Director of Web Technology for Future Sunday, 17 March 2013 Kicking unix servers, swearing at websites and running Ops teams for the last 15 years Particularly interested in culture of teams and organisations, so a lot of talk will be about that, why we were in hell, and how we got out - from Ops perspective because that’s where I come from

Future at a glance We have over 50m unique users from across the globe visiting our web properties monthly Online 50 million = 50,000 +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ +++++++++++++++++++++++++ ++++++++++++++++++++ +++++++++++++++++++ ++++++++++++++++++ +++++++ ++++++ +++++ Over $1m gross revenue per month via all digital editions. Over 3m paid iPad editions sold in FY12 Tablets = $10,000 = 200,000 $1 million Future has 225 licensed properties available in 89 overseas partnerships Licenses = 1 License 225 = 200,000 = 1 purchase Over 24m printed copies sold in )<²WKDW·V45 every minute of every day Print 24 million 1 Last year our content had over 120m YLGHRYLHZV²WKDW·V228 every minute of every day Video = 10 million = 10 views 120 million 10m 10m 10m 10m 10m 10m 10m 10m 10m 10m 10m 10m 10m 1 Future has 4.1m followers across all the main social networks Social Media 2.5m 1.3m = 200,000 300k 4.1 million Sunday, 17 March 2013 Future have been publishing for a long time 28 years in print publishing and now digital - websites and digital editions Future specialise in content for enthusiasts - if you have a hobby, we probably make a magazine and website for it Operations in UK, US and Australia 50 million unique users, 300 million page impressions a month The nature of publishing leads to a company structure that’s not like the average web company - publishers/mag structure/lots of little silos

A thousand different platforms Sunday, 17 March 2013 Ok, I lied a bit in the title, there are about 40 platforms. It just feels like a thousand different platforms when you try and run it all :) 80 major sites, 200+ smaller Different languages, dev teams, offices, countries One ops team

LIFE IN HELL Sunday, 17 March 2013 Manually : Manual config, manual deploys, manual rollbacks, manual pulling your hair out while sobbing in the corner and rocking Still managed to build some very successful websites like this Doing 250 deploys a month is nothing in the land of continuous deployment, but by hand it’s a hell of a lot of work : Over 10 deploys a day, failures, rollbacks, database changes. 2 people’s full time jobs! Hence the sobbing What else makes it stressful: Geographical and cultural separation between devs and ops Geographical separation makes all of the bad bits worse. All of the worst things you’d expect - silos, throw-it-over-the wall deploys, Blamestorming Ops tend to see devs as careless destroyers of their shiny servers Devs see Ops as grumpy old unix sysadmins who never want anything to change ever The business just wants the sites to get better more quickly and doesn’t want to referee

LIFE IN HELL OPERATIONS Sunday, 17 March 2013 Manually : Manual config, manual deploys, manual rollbacks, manual pulling your hair out while sobbing in the corner and rocking Still managed to build some very successful websites like this Doing 250 deploys a month is nothing in the land of continuous deployment, but by hand it’s a hell of a lot of work : Over 10 deploys a day, failures, rollbacks, database changes. 2 people’s full time jobs! Hence the sobbing What else makes it stressful: Geographical and cultural separation between devs and ops Geographical separation makes all of the bad bits worse. All of the worst things you’d expect - silos, throw-it-over-the wall deploys, Blamestorming Ops tend to see devs as careless destroyers of their shiny servers Devs see Ops as grumpy old unix sysadmins who never want anything to change ever The business just wants the sites to get better more quickly and doesn’t want to referee

LIFE IN HELL Sunday, 17 March 2013 Manually : Manual config, manual deploys, manual rollbacks, manual pulling your hair out while sobbing in the corner and rocking Still managed to build some very successful websites like this Doing 250 deploys a month is nothing in the land of continuous deployment, but by hand it’s a hell of a lot of work : Over 10 deploys a day, failures, rollbacks, database changes. 2 people’s full time jobs! Hence the sobbing What else makes it stressful: Geographical and cultural separation between devs and ops Geographical separation makes all of the bad bits worse. All of the worst things you’d expect - silos, throw-it-over-the wall deploys, Blamestorming Ops tend to see devs as careless destroyers of their shiny servers Devs see Ops as grumpy old unix sysadmins who never want anything to change ever The business just wants the sites to get better more quickly and doesn’t want to referee

Partners not parents Sunday, 17 March 2013 Another source of stress was being parental: Stereotypical Dev/Ops interactions can be like bad parent/child interactions: “Do as I tell you, you’re not to be trusted” “I hate you and you smell and I’m going to do it anyway” “Well now you’ve broken everything. I’m not angry, I’m just very disappointed. I’m going to go and clean everything up after you now so you can do it all again” This is dysfunctional! Cultural norms for Ops folk can be very unhelpful. BoFH is supposed to be comedy, not a manual for interpersonal relationships at work Parental cleanup doesn’t encourage anyone to own their own messes and fix them Being a parent is exhausting and doesn’t scale

Partners not parents THIS IS DYSFUNCTIONAL Sunday, 17 March 2013 Another source of stress was being parental: Stereotypical Dev/Ops interactions can be like bad parent/child interactions: “Do as I tell you, you’re not to be trusted” “I hate you and you smell and I’m going to do it anyway” “Well now you’ve broken everything. I’m not angry, I’m just very disappointed. I’m going to go and clean everything up after you now so you can do it all again” This is dysfunctional! Cultural norms for Ops folk can be very unhelpful. BoFH is supposed to be comedy, not a manual for interpersonal relationships at work Parental cleanup doesn’t encourage anyone to own their own messes and fix them Being a parent is exhausting and doesn’t scale

Highway to hell Sunday, 17 March 2013 So, what to do to fix it Things not to try: Adding more process Adding more rules Adding more restrictions e.g. complex signoffs, deploy windows, heavy process Like the legions of the damned trying to improve their lot by asking for better, sharper pitchforks to be used on them We probably tried all of these at one time or another. They universally made things worse. The tighter the controls, the worse everyone’s interactions were. Illusion of Control

Highway to hell Sunday, 17 March 2013 So, what to do to fix it Things not to try: Adding more process Adding more rules Adding more restrictions e.g. complex signoffs, deploy windows, heavy process Like the legions of the damned trying to improve their lot by asking for better, sharper pitchforks to be used on them We probably tried all of these at one time or another. They universally made things worse. The tighter the controls, the worse everyone’s interactions were. Illusion of Control

DevOps to the rescue? Sunday, 17 March 2013 We looked for an answer, DevOps sounded great, but info seemed to kind of assume you were working on a single platform, at a VC funded startup in the heart of SF Strangely, the company wasn’t keen to relocate us all there, so we had to find some other answers Popular Devops tools, processes and patterns weren’t a natural fit for us Hard to find business case to justify big bang to new structures and processes with all of the hit to schedules it would involve

Stairway to heaven Sunday, 17 March 2013 So what are the vital parts of DevOps that we wanted to adopt. The core for us was to recognise that DevOps was about: People Collaborating together On the same goals: Deliver more value to the business, more quickly, and actually be able to sleep at night

Management buy-in to DevOps Sunday, 17 March 2013 It’s not easy to sell DevOps to management from a cold start If we couldn’t sell it to the technical teams, then it would be impossible to sell to the business So many different teams/management, hard to impose from above Businesses are (mostly) rational - if we could show how it really helped, it’s a much easier sell

Management buy-in to DevOps • “Itʼs easier to ask forgiveness than to get permission” Sunday, 17 March 2013 It’s not easy to sell DevOps to management from a cold start If we couldn’t sell it to the technical teams, then it would be impossible to sell to the business So many different teams/management, hard to impose from above Businesses are (mostly) rational - if we could show how it really helped, it’s a much easier sell

Management buy-in to DevOps • Itʼs easier to demonstrate business value after youʼve generated some • Bottom-up adoption is easier to drive than top-down • “Itʼs easier to ask forgiveness than to get permission” Sunday, 17 March 2013 It’s not easy to sell DevOps to management from a cold start If we couldn’t sell it to the technical teams, then it would be impossible to sell to the business So many different teams/management, hard to impose from above Businesses are (mostly) rational - if we could show how it really helped, it’s a much easier sell

Jacobʼs ladder Sunday, 17 March 2013 So how to get out of hell - apart from “Well I wouldn’t start from here if I were you”? Plan of attack Accept things will break, that’s why you start small, breaking things ‘narrows the problem space’ Earliest adopters were ops, ‘eat your own dogfood’ We tested a lot of the toolset and ideas on our own puppet stuff Pick things that can be solved and will really improve things for people Evolve solutions piecemeal with Dev and Ops together Make a toolbox of things for other people to use and build on

Jacobʼs ladder • Start small • Find some early adopters • Eat your own dogfood Sunday, 17 March 2013 So how to get out of hell - apart from “Well I wouldn’t start from here if I were you”? Plan of attack Accept things will break, that’s why you start small, breaking things ‘narrows the problem space’ Earliest adopters were ops, ‘eat your own dogfood’ We tested a lot of the toolset and ideas on our own puppet stuff Pick things that can be solved and will really improve things for people Evolve solutions piecemeal with Dev and Ops together Make a toolbox of things for other people to use and build on

Jacobʼs ladder • Start small • Find some early adopters • Eat your own dogfood • Find solvable problems • Form partnerships • Evolve solutions together to make a toolbox Sunday, 17 March 2013 So how to get out of hell - apart from “Well I wouldn’t start from here if I were you”? Plan of attack Accept things will break, that’s why you start small, breaking things ‘narrows the problem space’ Earliest adopters were ops, ‘eat your own dogfood’ We tested a lot of the toolset and ideas on our own puppet stuff Pick things that can be solved and will really improve things for people Evolve solutions piecemeal with Dev and Ops together Make a toolbox of things for other people to use and build on

Hell on wheels Sunday, 17 March 2013 Finding people who really wanted something different was the key to making this work Who do you pick for DevOps? Cyclists! - cyclingnews.com, bikeradar.com Mature platform, enthusiastic developers, supportive management What we had to offer: More control for the devs Quicker delivery for management Less pain for Operations Started working with them to improve both culture and toolset

Hell on wheels Sunday, 17 March 2013 Finding people who really wanted something different was the key to making this work Who do you pick for DevOps? Cyclists! - cyclingnews.com, bikeradar.com Mature platform, enthusiastic developers, supportive management What we had to offer: More control for the devs Quicker delivery for management Less pain for Operations Started working with them to improve both culture and toolset

Hell on wheels Sunday, 17 March 2013 Finding people who really wanted something different was the key to making this work Who do you pick for DevOps? Cyclists! - cyclingnews.com, bikeradar.com Mature platform, enthusiastic developers, supportive management What we had to offer: More control for the devs Quicker delivery for management Less pain for Operations Started working with them to improve both culture and toolset

Cultural changes Sunday, 17 March 2013 Biggest and most effective changes are about communication and teamwork Ops had to learn to ‘relax and let go’ of the infrastructure etc. Knowledge sharing Sitting in the same place helps enormously, but good use of IRC or similar really helps with geographically separated teams Failcake - blame free post mortem encouragement If people cause breakages that affect others, be honest, own up, and bring cake. The situation is discussed, but no- one is angry, because, hey, there’s cake!

Cultural changes • Openness • Shared responsibility • Cross-departmental meetings • Lunch and learn Sunday, 17 March 2013 Biggest and most effective changes are about communication and teamwork Ops had to learn to ‘relax and let go’ of the infrastructure etc. Knowledge sharing Sitting in the same place helps enormously, but good use of IRC or similar really helps with geographically separated teams Failcake - blame free post mortem encouragement If people cause breakages that affect others, be honest, own up, and bring cake. The situation is discussed, but no- one is angry, because, hey, there’s cake!

Cultural changes • Openness • Shared responsibility • Cross-departmental meetings • Lunch and learn • Sitting in the same office! • IRC Sunday, 17 March 2013 Biggest and most effective changes are about communication and teamwork Ops had to learn to ‘relax and let go’ of the infrastructure etc. Knowledge sharing Sitting in the same place helps enormously, but good use of IRC or similar really helps with geographically separated teams Failcake - blame free post mortem encouragement If people cause breakages that affect others, be honest, own up, and bring cake. The situation is discussed, but no- one is angry, because, hey, there’s cake!

Cultural changes • Openness • Shared responsibility • Cross-departmental meetings • Lunch and learn • Failcake • Sitting in the same office! • IRC Sunday, 17 March 2013 Biggest and most effective changes are about communication and teamwork Ops had to learn to ‘relax and let go’ of the infrastructure etc. Knowledge sharing Sitting in the same place helps enormously, but good use of IRC or similar really helps with geographically separated teams Failcake - blame free post mortem encouragement If people cause breakages that affect others, be honest, own up, and bring cake. The situation is discussed, but no- one is angry, because, hey, there’s cake!

The toolbox Sunday, 17 March 2013 We’ve used a lot of the usual culprits, ActiveMQ is worth picking out because of the the effect it’s had on us technically, but also culturally

The toolbox • Puppet • MCollective • ActiveMQ • Graphite • Statsd • Vagrant • Puppet Environments • Git • Liquibase • Jenkins Sunday, 17 March 2013 We’ve used a lot of the usual culprits, ActiveMQ is worth picking out because of the the effect it’s had on us technically, but also culturally

Message queues are magic glue Sunday, 17 March 2013 MCollective meant we accidentally gained a message queue ActiveMQ for us, but any queue will do really Once we had that on every server, we realised just how powerful it was, and how many new things you could build on that base Makes building bespoke tools out of distributed components easier Makes different teams writing tools that work together much easier

Message queues are magic glue • Accidentally a whole ActiveMQ Sunday, 17 March 2013 MCollective meant we accidentally gained a message queue ActiveMQ for us, but any queue will do really Once we had that on every server, we realised just how powerful it was, and how many new things you could build on that base Makes building bespoke tools out of distributed components easier Makes different teams writing tools that work together much easier

Message queues are magic glue • Accidentally a whole ActiveMQ • One standard way for everything to communicate Sunday, 17 March 2013 MCollective meant we accidentally gained a message queue ActiveMQ for us, but any queue will do really Once we had that on every server, we realised just how powerful it was, and how many new things you could build on that base Makes building bespoke tools out of distributed components easier Makes different teams writing tools that work together much easier

Message queues are magic glue • Accidentally a whole ActiveMQ • One standard way for everything to communicate • Makes building bespoke tools out of smaller distributed components easier Sunday, 17 March 2013 MCollective meant we accidentally gained a message queue ActiveMQ for us, but any queue will do really Once we had that on every server, we realised just how powerful it was, and how many new things you could build on that base Makes building bespoke tools out of distributed components easier Makes different teams writing tools that work together much easier

Message queues are magic glue • Accidentally a whole ActiveMQ • One standard way for everything to communicate • Makes building bespoke tools out of smaller distributed components easier • Makes different teams writing tools that work together much easier Sunday, 17 March 2013 MCollective meant we accidentally gained a message queue ActiveMQ for us, but any queue will do really Once we had that on every server, we realised just how powerful it was, and how many new things you could build on that base Makes building bespoke tools out of distributed components easier Makes different teams writing tools that work together much easier

Future DeployKit ActiveMQ 1a2b3c Git Server STUFF HAPPENED OMGWTFBBQ EventBot on IRC Commit Made Events Sunday, 17 March 2013 John Hawkes-Reed did most of the work - mistakes in explanation are mine! Every push to the git repo triggers a post-commit hook that sends a message to a topic Subscribers to the topic get the message and do things Simplest subscriber is Eventbot: tells everyone on IRC about the events

Future DeployKit ActiveMQ 1a2b3c Git Server Jenkins STUFF HAPPENED OMGWTFBBQ EventBot on IRC Commit Made Tests Passed Events Commit Made Sunday, 17 March 2013 Jenkins: Gets message from the queue Gets its copy of the repo in sync Runs unit tests/build/whatever

Future DeployKit ActiveMQ 1a2b3c Git Server Jenkins STUFF HAPPENED OMGWTFBBQ EventBot on IRC Commit Made Tests Passed Events Commit Made Tests Passed Tests Passed Tests Passed Sunday, 17 March 2013 Message back to queue about success or failure - in this case success

Future DeployKit ActiveMQ 1a2b3c Git Server Jenkins 1a2b3c STUFF HAPPENED OMGWTFBBQ EventBot on IRC Commit Made Tests Passed Events Commit Made Tests Passed Tests Passed Tests Passed Sunday, 17 March 2013 Servers with puppet/heira data that says they have that codebase do a git fetch Update local repo Means weʼre not dependent on the git master for deploys

Future DeployKit ActiveMQ STUFF HAPPENED OMGWTFBBQ EventBot on IRC Events Deploy TR-471 Sunday, 17 March 2013 For deploys, we have a thin web interface wrapping an MCollective client and agent Developers use web interface to pick what tag to deploy where - stage/live/whatever

Future DeployKit ActiveMQ STUFF HAPPENED OMGWTFBBQ EventBot on IRC Events Deploy Success Deploy TR-471 Deploy TR-471 Run pre-deploy script Checkout git work tree Run post-deploy script Sunday, 17 March 2013 MCollective agent triggers the deployment process on appropriate servers Deploy is all from the local git repo Predeploy script runs, sanity checks, whatever else is needed, if it passes ok Deploy - by checking out the git repo using the --work-tree flag to deploy to right place without git metadata in the dirtree Postdeploy script - database migrations, post deploy cleanup, etc Signal success back to the queue

Future DeployKit ActiveMQ STUFF HAPPENED OMGWTFBBQ EventBot on IRC Events Deploy Success Deploy TR-471 Deploy TR-471 Sunday, 17 March 2013 Rinse and repeat on the other servers. Weʼre doing them one at a time, but MCollective lets you do them all in parallel, in batches, with delays between them, itʼs very versatile Topic subscribers mark deploy in metrics, send messages to stakeholders, report in IRC, anything you want really Anyone can add other listeners and do things based on these events We ended up throwing pretty much everything onto the queue, and this had more interesting cultural effects

Radical transparency Sunday, 17 March 2013 Event driven architectures change culture: mostly because of visibility Radical transparency - commits/deploys/metrics/failures/etc Make it harder to hide mistakes Make responsibility more shared Reduces 'because I say so' and makes it a discussion about a visible, shared bit of info Less siloing because it avoids everyone having their own special snowflake of every service Helps management : shows the business which sites/behaviour/etc are impacting on others People make interesting things you don’t anticipate once they have access to this kind of information and common toolset

Ownership and toolmaking Sunday, 17 March 2013 Another cultural change is about ownership: The things you ‘own’ own you 'Owning things' - being the 'person who does X' is self-siloing The mail admin, the dba, etc Not useful Want to be toolmakers, not tools :) So must let go more Ops stop being the mystical holders of the information about the servers Transparency/Visibility again Inverting organisation - guts on display Solidarity

Anarchy in the UK Sunday, 17 March 2013 Solidarity you say? Should we all join the Socialist Workers Party? Again, JHR piped up Anarcho-syndicalism Principles of AS - 1) Workers' self-management, 2) Direct Action, 3) Workers' solidarity Good principles for DevOps teams

Seven useful sins Sunday, 17 March 2013 Roads out of Hell - we found that there were some useful ‘sins’ that helped adoption: (credit to the Perl virtues here really) Envy: why do their deploys take less time than ours? why is their site more reliable? Pride: People want their projects to be better than other peoples Sloth: Better use of the tools and processes means less boring work for more effect Greed: How do I make my project more profitable? Gluttony: Failcake is really tasty, but fattening ;) Wrath: People are angry and frustrated when things don’t work, direct that at making more useful changes Lust: Must maintain a good work/life balance, so saving at least one sin for outside work is good Are we out of Hell yet?

Purgatory Sunday, 17 March 2013 Purgatory: the state or place of purification and temporary punishment on the way to Heaven How has DevOps been adopted, and how has it affected the business? Central, cross-functional teams Build services/APIs Search Mobile sites User management Commenting Etc Provide different tools with same principles - svn deployer Corp Java - came for the message queues, got git, jenkins, RESTful services etc

Revelation _ *"_"* __ /`_`\ __ .' '. | / \ | .' '. , / ')\^_^/(' \ , \`--' . (_.> <._) . '--`/ '.__.' '._/ \_/ \_.' '.__.' / , _ , \ \ \_/|\_/ / \ //^\\ / \/` `\/ | | | | | | | | .. ..:::.| | ..::::. .. ..::::..::::... .::::::::| |:::::::::::::::. ::::::::::::::::::.:::::::::::| |:::::::::::::::::. ':::::::::::::::::::::::::::::| |::::::::::::::::::: ::::::::::::::::::::::::::::| |::::::::::::::::::' '':::' '::::::::::::::::\_.__./:::::::::::::::'' '':::::::::::::::::::::::::::::::::' jgs '::::::::::::::::::::::::::::::'' '':::::'' '''::::::'' Sunday, 17 March 2013 So, with the visible successes in place, it was much easier to approach the wider business about making these kind of changes, and we do that by Business needs: More content, More functionality, More quickly, More reliably: More revenue This is what DevOps adoption offers them We were able to show how the cultural and technical changes from devops practices provide what the business wants, not just technical teams Radical transparency makes the costs and benefits visible to all Don’t have to drive adoption so much: The developers on other sites wanted the same tools and privileges The project managers and publishers wanted the same increase in speed and decrease in cost Evangelising is done for you by the people you’d otherwise have to persuade to try it

Revelation • Understand the goals of the business _ *"_"* __ /`_`\ __ .' '. | / \ | .' '. , / ')\^_^/(' \ , \`--' . (_.> <._) . '--`/ '.__.' '._/ \_/ \_.' '.__.' / , _ , \ \ \_/|\_/ / \ //^\\ / \/` `\/ | | | | | | | | .. ..:::.| | ..::::. .. ..::::..::::... .::::::::| |:::::::::::::::. ::::::::::::::::::.:::::::::::| |:::::::::::::::::. ':::::::::::::::::::::::::::::| |::::::::::::::::::: ::::::::::::::::::::::::::::| |::::::::::::::::::' '':::' '::::::::::::::::\_.__./:::::::::::::::'' '':::::::::::::::::::::::::::::::::' jgs '::::::::::::::::::::::::::::::'' '':::::'' '''::::::'' Sunday, 17 March 2013 So, with the visible successes in place, it was much easier to approach the wider business about making these kind of changes, and we do that by Business needs: More content, More functionality, More quickly, More reliably: More revenue This is what DevOps adoption offers them We were able to show how the cultural and technical changes from devops practices provide what the business wants, not just technical teams Radical transparency makes the costs and benefits visible to all Don’t have to drive adoption so much: The developers on other sites wanted the same tools and privileges The project managers and publishers wanted the same increase in speed and decrease in cost Evangelising is done for you by the people you’d otherwise have to persuade to try it

Revelation • Understand the goals of the business • Demonstrate how teams that make these cultural and technical changes deliver these goals _ *"_"* __ /`_`\ __ .' '. | / \ | .' '. , / ')\^_^/(' \ , \`--' . (_.> <._) . '--`/ '.__.' '._/ \_/ \_.' '.__.' / , _ , \ \ \_/|\_/ / \ //^\\ / \/` `\/ | | | | | | | | .. ..:::.| | ..::::. .. ..::::..::::... .::::::::| |:::::::::::::::. ::::::::::::::::::.:::::::::::| |:::::::::::::::::. ':::::::::::::::::::::::::::::| |::::::::::::::::::: ::::::::::::::::::::::::::::| |::::::::::::::::::' '':::' '::::::::::::::::\_.__./:::::::::::::::'' '':::::::::::::::::::::::::::::::::' jgs '::::::::::::::::::::::::::::::'' '':::::'' '''::::::'' Sunday, 17 March 2013 So, with the visible successes in place, it was much easier to approach the wider business about making these kind of changes, and we do that by Business needs: More content, More functionality, More quickly, More reliably: More revenue This is what DevOps adoption offers them We were able to show how the cultural and technical changes from devops practices provide what the business wants, not just technical teams Radical transparency makes the costs and benefits visible to all Don’t have to drive adoption so much: The developers on other sites wanted the same tools and privileges The project managers and publishers wanted the same increase in speed and decrease in cost Evangelising is done for you by the people you’d otherwise have to persuade to try it

Revelation • Understand the goals of the business • Demonstrate how teams that make these cultural and technical changes deliver these goals • Make the costs and benefits transparent _ *"_"* __ /`_`\ __ .' '. | / \ | .' '. , / ')\^_^/(' \ , \`--' . (_.> <._) . '--`/ '.__.' '._/ \_/ \_.' '.__.' / , _ , \ \ \_/|\_/ / \ //^\\ / \/` `\/ | | | | | | | | .. ..:::.| | ..::::. .. ..::::..::::... .::::::::| |:::::::::::::::. ::::::::::::::::::.:::::::::::| |:::::::::::::::::. ':::::::::::::::::::::::::::::| |::::::::::::::::::: ::::::::::::::::::::::::::::| |::::::::::::::::::' '':::' '::::::::::::::::\_.__./:::::::::::::::'' '':::::::::::::::::::::::::::::::::' jgs '::::::::::::::::::::::::::::::'' '':::::'' '''::::::'' Sunday, 17 March 2013 So, with the visible successes in place, it was much easier to approach the wider business about making these kind of changes, and we do that by Business needs: More content, More functionality, More quickly, More reliably: More revenue This is what DevOps adoption offers them We were able to show how the cultural and technical changes from devops practices provide what the business wants, not just technical teams Radical transparency makes the costs and benefits visible to all Don’t have to drive adoption so much: The developers on other sites wanted the same tools and privileges The project managers and publishers wanted the same increase in speed and decrease in cost Evangelising is done for you by the people you’d otherwise have to persuade to try it

Revelation • Understand the goals of the business • Demonstrate how teams that make these cultural and technical changes deliver these goals • Make the costs and benefits transparent • Deliver first, then evangelise _ *"_"* __ /`_`\ __ .' '. | / \ | .' '. , / ')\^_^/(' \ , \`--' . (_.> <._) . '--`/ '.__.' '._/ \_/ \_.' '.__.' / , _ , \ \ \_/|\_/ / \ //^\\ / \/` `\/ | | | | | | | | .. ..:::.| | ..::::. .. ..::::..::::... .::::::::| |:::::::::::::::. ::::::::::::::::::.:::::::::::| |:::::::::::::::::. ':::::::::::::::::::::::::::::| |::::::::::::::::::: ::::::::::::::::::::::::::::| |::::::::::::::::::' '':::' '::::::::::::::::\_.__./:::::::::::::::'' '':::::::::::::::::::::::::::::::::' jgs '::::::::::::::::::::::::::::::'' '':::::'' '''::::::'' Sunday, 17 March 2013 So, with the visible successes in place, it was much easier to approach the wider business about making these kind of changes, and we do that by Business needs: More content, More functionality, More quickly, More reliably: More revenue This is what DevOps adoption offers them We were able to show how the cultural and technical changes from devops practices provide what the business wants, not just technical teams Radical transparency makes the costs and benefits visible to all Don’t have to drive adoption so much: The developers on other sites wanted the same tools and privileges The project managers and publishers wanted the same increase in speed and decrease in cost Evangelising is done for you by the people you’d otherwise have to persuade to try it

Sunday, 17 March 2013 So that’s how we started to work our way out of hell It’s not finished yet, there’s still lots of work to do It’s not an easy road It starts from the bottom of the pit, and the only way is upwards For us it was about the culture and the people Give people the chance to try something better and see it working for others Give them the tools and flexibility to climb out themselves Let the early adopters lead the rest

Thank you • Source for our tools: https://github.com/ FuturePublishing/ • Future: http://futureplc.com/ • Future DevOps Blog: http://ops.failcake.net/ • Contact me: sam@fqdn.net, @thesamoth Sunday, 17 March 2013 To help others out of hell, we’ve open sourced a lot of the tools we’ve built along the road I hope seeing some of our journey has been useful for you Thank you - Any questions?